{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "pycharm-2c75a13c",
      "language": "python",
      "display_name": "PyCharm (FinRL)"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
    },
    "colab": {
      "name": "FinRL_PortfolioAllocation_NeurIPS_2020.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "lvrqTro3lhAh",
        "a3Iuv554xYFH",
        "SPEXBcm-uBJo",
        "iidB5E27dfzh"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AI4Finance-Foundation/FinRL/blob/master/FinRL_PortfolioAllocation_NeurIPS_2020.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yv3IDvrobU37"
      },
      "source": [
        "# Deep Reinforcement Learning for Stock Trading from Scratch: Portfolio Allocation\n",
        "\n",
        "Tutorials to use OpenAI DRL to perform portfolio allocation in one Jupyter Notebook | Presented at NeurIPS 2020: Deep RL Workshop\n",
        "\n",
        "* This blog is based on our paper: FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance, presented at NeurIPS 2020: Deep RL Workshop.\n",
        "* Check out medium blog for detailed explanations: \n",
        "* Please report any issues to our Github: https://github.com/AI4Finance-Foundation/FinRL/issues\n",
        "* **Pytorch Version** \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4kHCfEiTA80V"
      },
      "source": [
        "# Content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUmLTmoQA7_w"
      },
      "source": [
        "* [1. Problem Definition](#0)\n",
        "* [2. Getting Started - Load Python packages](#1)\n",
        "    * [2.1. Install Packages](#1.1)    \n",
        "    * [2.2. Check Additional Packages](#1.2)\n",
        "    * [2.3. Import Packages](#1.3)\n",
        "    * [2.4. Create Folders](#1.4)\n",
        "* [3. Download Data](#2)\n",
        "* [4. Preprocess Data](#3)        \n",
        "    * [4.1. Technical Indicators](#3.1)\n",
        "    * [4.2. Perform Feature Engineering](#3.2)\n",
        "* [5.Build Environment](#4)  \n",
        "    * [5.1. Training & Trade Data Split](#4.1)\n",
        "    * [5.2. User-defined Environment](#4.2)   \n",
        "    * [5.3. Initialize Environment](#4.3)    \n",
        "* [6.Implement DRL Algorithms](#5)  \n",
        "* [7.Backtesting Performance](#6)  \n",
        "    * [7.1. BackTestStats](#6.1)\n",
        "    * [7.2. BackTestPlot](#6.2)   \n",
        "    * [7.3. Baseline Stats](#6.3)   \n",
        "    * [7.3. Compare to Stock Market Index](#6.4)             "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12v1i0jVkg48"
      },
      "source": [
        "<a id='0'></a>\n",
        "# Part 1. Problem Definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L63HKnWvkirx"
      },
      "source": [
        "This problem is to design an automated trading solution for portfolio alloacation. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.\n",
        "\n",
        "The algorithm is trained using Deep Reinforcement Learning (DRL) algorithms and the components of the reinforcement learning environment are:\n",
        "\n",
        "\n",
        "* Action: The action space describes the allowed actions that the agent interacts with the\n",
        "environment. Normally, a ∈ A represents the weight of a stock in the porfolio: a ∈ (-1,1). Assume our stock pool includes N stocks, we can use a list [a<sub>1</sub>, a<sub>2</sub>, ... , a<sub>N</sub>] to determine the weight for each stock in the porfotlio, where a<sub>i</sub> ∈ (-1,1), a<sub>1</sub>+ a<sub>2</sub>+...+a<sub>N</sub>=1. For example, \"The weight of AAPL in the portfolio is 10%.\" is [0.1 , ...].\n",
        "\n",
        "* Reward function: r(s, a, s′) is the incentive mechanism for an agent to learn a better action. The change of the portfolio value when action a is taken at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio\n",
        "values at state s′ and s, respectively\n",
        "\n",
        "* State: The state space describes the observations that the agent receives from the environment. Just as a human trader needs to analyze various information before executing a trade, so\n",
        "our trading agent observes many different features to better learn in an interactive environment.\n",
        "\n",
        "* Environment: Dow 30 consituents\n",
        "\n",
        "\n",
        "The data of the single stock that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g_emqQCCklVt"
      },
      "source": [
        "<a id='1'></a>\n",
        "# Part 2. Getting Started- Load Python Packages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cVCcCalAknGn"
      },
      "source": [
        "<a id='1.1'></a>\n",
        "## 2.1. Install all the packages through FinRL library\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pT8a0fvhA_TW",
        "outputId": "1eba9a59-ff73-47c1-bfd9-d32ea43c0a5c",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "## install finrl library\n",
        "!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting git+https://github.com/AI4Finance-LLC/FinRL-Library.git\n",
            "  Cloning https://github.com/AI4Finance-LLC/FinRL-Library.git to /tmp/pip-req-build-oipawpb8\n",
            "  Running command git clone -q https://github.com/AI4Finance-LLC/FinRL-Library.git /tmp/pip-req-build-oipawpb8\n",
            "Collecting pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2\n",
            "  Cloning https://github.com/quantopian/pyfolio.git to /tmp/pip-install-4ptpp4pi/pyfolio_377389ebaaf94ca2ae8ea34ffec2bbbb\n",
            "  Running command git clone -q https://github.com/quantopian/pyfolio.git /tmp/pip-install-4ptpp4pi/pyfolio_377389ebaaf94ca2ae8ea34ffec2bbbb\n",
            "Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl\n",
            "  Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to /tmp/pip-install-4ptpp4pi/elegantrl_50a0aac9daf4450382788898004faabc\n",
            "  Running command git clone -q https://github.com/AI4Finance-Foundation/ElegantRL.git /tmp/pip-install-4ptpp4pi/elegantrl_50a0aac9daf4450382788898004faabc\n",
            "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (1.19.5)\n",
            "Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (1.1.5)\n",
            "Collecting stockstats>=0.4.0\n",
            "  Downloading stockstats-0.4.1-py2.py3-none-any.whl (19 kB)\n",
            "Collecting yfinance\n",
            "  Downloading yfinance-0.1.69-py2.py3-none-any.whl (26 kB)\n",
            "Collecting elegantrl\n",
            "  Downloading elegantrl-0.3.3-py3-none-any.whl (234 kB)\n",
            "\u001b[K     |████████████████████████████████| 234 kB 6.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (3.2.2)\n",
            "Requirement already satisfied: scikit-learn>=0.21.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (1.0.2)\n",
            "Requirement already satisfied: gym>=0.17 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (0.17.3)\n",
            "Collecting stable-baselines3[extra]\n",
            "  Downloading stable_baselines3-1.3.0-py3-none-any.whl (174 kB)\n",
            "\u001b[K     |████████████████████████████████| 174 kB 45.7 MB/s \n",
            "\u001b[?25hCollecting ray[default]\n",
            "  Downloading ray-1.9.2-cp37-cp37m-manylinux2014_x86_64.whl (57.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 57.6 MB 94 kB/s \n",
            "\u001b[?25hCollecting lz4\n",
            "  Downloading lz4-3.1.10-cp37-cp37m-manylinux2010_x86_64.whl (1.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.8 MB 20.8 MB/s \n",
            "\u001b[?25hCollecting tensorboardX\n",
            "  Downloading tensorboardX-2.4.1-py2.py3-none-any.whl (124 kB)\n",
            "\u001b[K     |████████████████████████████████| 124 kB 58.7 MB/s \n",
            "\u001b[?25hCollecting gputil\n",
            "  Downloading GPUtil-1.4.0.tar.gz (5.5 kB)\n",
            "Collecting exchange_calendars\n",
            "  Downloading exchange_calendars-3.5.tar.gz (147 kB)\n",
            "\u001b[K     |████████████████████████████████| 147 kB 66.8 MB/s \n",
            "\u001b[?25hCollecting alpaca_trade_api\n",
            "  Downloading alpaca_trade_api-1.4.3-py3-none-any.whl (36 kB)\n",
            "Collecting ccxt>=1.66.32\n",
            "  Downloading ccxt-1.67.31-py2.py3-none-any.whl (2.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.3 MB 38.1 MB/s \n",
            "\u001b[?25hCollecting jqdatasdk\n",
            "  Downloading jqdatasdk-1.8.10-py3-none-any.whl (153 kB)\n",
            "\u001b[K     |████████████████████████████████| 153 kB 60.7 MB/s \n",
            "\u001b[?25hCollecting wrds\n",
            "  Downloading wrds-3.1.1-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (3.6.4)\n",
            "Requirement already satisfied: setuptools>=41.4.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (57.4.0)\n",
            "Requirement already satisfied: wheel>=0.33.6 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.4) (0.37.1)\n",
            "Collecting pre-commit\n",
            "  Downloading pre_commit-2.16.0-py2.py3-none-any.whl (191 kB)\n",
            "\u001b[K     |████████████████████████████████| 191 kB 60.4 MB/s \n",
            "\u001b[?25hCollecting pybullet\n",
            "  Downloading pybullet-3.2.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (90.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 90.8 MB 2.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.4) (1.10.0+cu111)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.4) (4.1.2.30)\n",
            "Collecting box2d-py\n",
            "  Downloading box2d_py-2.3.8-cp37-cp37m-manylinux1_x86_64.whl (448 kB)\n",
            "\u001b[K     |████████████████████████████████| 448 kB 76.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (5.5.0)\n",
            "Requirement already satisfied: pytz>=2014.10 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (2018.9)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (1.4.1)\n",
            "Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.11.2)\n",
            "Collecting empyrical>=0.5.0\n",
            "  Downloading empyrical-0.5.5.tar.gz (52 kB)\n",
            "\u001b[K     |████████████████████████████████| 52 kB 1.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: certifi>=2018.1.18 in /usr/local/lib/python3.7/dist-packages (from ccxt>=1.66.32->finrl==0.3.4) (2021.10.8)\n",
            "Requirement already satisfied: requests>=2.18.4 in /usr/local/lib/python3.7/dist-packages (from ccxt>=1.66.32->finrl==0.3.4) (2.23.0)\n",
            "Collecting yarl==1.7.2\n",
            "  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
            "\u001b[K     |████████████████████████████████| 271 kB 61.9 MB/s \n",
            "\u001b[?25hCollecting cryptography>=2.6.1\n",
            "  Downloading cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.6 MB 43.5 MB/s \n",
            "\u001b[?25hCollecting aiodns>=1.1.1\n",
            "  Downloading aiodns-3.0.0-py3-none-any.whl (5.0 kB)\n",
            "Collecting aiohttp>=3.8\n",
            "  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 59.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from yarl==1.7.2->ccxt>=1.66.32->finrl==0.3.4) (3.10.0.2)\n",
            "Collecting multidict>=4.0\n",
            "  Downloading multidict-5.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)\n",
            "\u001b[K     |████████████████████████████████| 160 kB 61.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl==1.7.2->ccxt>=1.66.32->finrl==0.3.4) (2.10)\n",
            "Collecting pycares>=4.0.0\n",
            "  Downloading pycares-4.1.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (291 kB)\n",
            "\u001b[K     |████████████████████████████████| 291 kB 62.9 MB/s \n",
            "\u001b[?25hCollecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (192 kB)\n",
            "\u001b[K     |████████████████████████████████| 192 kB 60.0 MB/s \n",
            "\u001b[?25hCollecting asynctest==0.13.0\n",
            "  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
            "Collecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)\n",
            "Collecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt>=1.66.32->finrl==0.3.4) (2.0.10)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt>=1.66.32->finrl==0.3.4) (21.4.0)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=2.6.1->ccxt>=1.66.32->finrl==0.3.4) (1.15.0)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt>=1.66.32->finrl==0.3.4) (2.21)\n",
            "Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.7/dist-packages (from empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.9.0)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.4) (1.5.0)\n",
            "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.4) (1.3.0)\n",
            "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (4.8.0)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (2.6.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (4.4.2)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (5.1.1)\n",
            "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (1.0.18)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.7.5)\n",
            "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.8.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.4) (2.8.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.4) (1.3.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.4) (3.0.6)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.4) (0.11.0)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (4.2.6)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.2.5)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (1.15.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym>=0.17->finrl==0.3.4) (0.16.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->ccxt>=1.66.32->finrl==0.3.4) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->ccxt>=1.66.32->finrl==0.3.4) (3.0.4)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.0->finrl==0.3.4) (1.1.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.0->finrl==0.3.4) (3.0.0)\n",
            "Collecting alpaca_trade_api\n",
            "  Downloading alpaca_trade_api-1.4.2-py3-none-any.whl (36 kB)\n",
            "  Downloading alpaca_trade_api-1.4.1-py3-none-any.whl (36 kB)\n",
            "  Downloading alpaca_trade_api-1.4.0-py3-none-any.whl (34 kB)\n",
            "  Downloading alpaca_trade_api-1.3.0-py3-none-any.whl (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 1.6 MB/s \n",
            "\u001b[?25h  Downloading alpaca_trade_api-1.2.3-py3-none-any.whl (40 kB)\n",
            "\u001b[K     |████████████████████████████████| 40 kB 5.6 MB/s \n",
            "\u001b[?25hCollecting websockets<10,>=8.0\n",
            "  Downloading websockets-9.1-cp37-cp37m-manylinux2010_x86_64.whl (103 kB)\n",
            "\u001b[K     |████████████████████████████████| 103 kB 47.1 MB/s \n",
            "\u001b[?25hCollecting websocket-client<2,>=0.56.0\n",
            "  Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)\n",
            "\u001b[K     |████████████████████████████████| 53 kB 1.8 MB/s \n",
            "\u001b[?25hCollecting msgpack==1.0.2\n",
            "  Downloading msgpack-1.0.2-cp37-cp37m-manylinux1_x86_64.whl (273 kB)\n",
            "\u001b[K     |████████████████████████████████| 273 kB 59.3 MB/s \n",
            "\u001b[?25hCollecting pyluach\n",
            "  Downloading pyluach-1.3.0-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from exchange_calendars->finrl==0.3.4) (0.11.2)\n",
            "Requirement already satisfied: korean_lunar_calendar in /usr/local/lib/python3.7/dist-packages (from exchange_calendars->finrl==0.3.4) (0.2.1)\n",
            "Collecting pymysql>=0.7.6\n",
            "  Downloading PyMySQL-1.0.2-py3-none-any.whl (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 2.0 MB/s \n",
            "\u001b[?25hCollecting thriftpy2>=0.3.9\n",
            "  Downloading thriftpy2-0.4.14.tar.gz (361 kB)\n",
            "\u001b[K     |████████████████████████████████| 361 kB 76.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: SQLAlchemy>=1.2.8 in /usr/local/lib/python3.7/dist-packages (from jqdatasdk->finrl==0.3.4) (1.4.29)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.4) (1.1.2)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.4) (4.10.0)\n",
            "Collecting ply<4.0,>=3.4\n",
            "  Downloading ply-3.11-py2.py3-none-any.whl (49 kB)\n",
            "\u001b[K     |████████████████████████████████| 49 kB 5.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.4) (3.7.0)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.4) (0.7.0)\n",
            "Collecting nodeenv>=0.11.1\n",
            "  Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)\n",
            "Collecting virtualenv>=20.0.8\n",
            "  Downloading virtualenv-20.13.0-py2.py3-none-any.whl (6.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.5 MB 37.8 MB/s \n",
            "\u001b[?25hCollecting cfgv>=2.0.0\n",
            "  Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)\n",
            "Collecting identify>=1.0.0\n",
            "  Downloading identify-2.4.3-py2.py3-none-any.whl (98 kB)\n",
            "\u001b[K     |████████████████████████████████| 98 kB 7.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.4) (0.10.2)\n",
            "Collecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 46.1 MB/s \n",
            "\u001b[?25hCollecting platformdirs<3,>=2\n",
            "  Downloading platformdirs-2.4.1-py3-none-any.whl (14 kB)\n",
            "Collecting distlib<1,>=0.3.1\n",
            "  Downloading distlib-0.3.4-py2.py3-none-any.whl (461 kB)\n",
            "\u001b[K     |████████████████████████████████| 461 kB 51.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock<4,>=3.2 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->finrl==0.3.4) (3.4.2)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.4) (0.7.1)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.4) (1.11.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.4) (1.4.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.4) (8.12.0)\n",
            "Requirement already satisfied: grpcio>=1.28.1 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (1.43.0)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (7.1.2)\n",
            "Requirement already satisfied: protobuf>=3.15.3 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (3.17.3)\n",
            "Collecting redis>=3.5.0\n",
            "  Downloading redis-4.1.0-py3-none-any.whl (171 kB)\n",
            "\u001b[K     |████████████████████████████████| 171 kB 61.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: jsonschema in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (4.3.3)\n",
            "Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (0.12.0)\n",
            "Requirement already satisfied: smart-open in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (5.2.1)\n",
            "Collecting colorful\n",
            "  Downloading colorful-0.5.4-py2.py3-none-any.whl (201 kB)\n",
            "\u001b[K     |████████████████████████████████| 201 kB 62.4 MB/s \n",
            "\u001b[?25hCollecting opencensus\n",
            "  Downloading opencensus-0.8.0-py2.py3-none-any.whl (128 kB)\n",
            "\u001b[K     |████████████████████████████████| 128 kB 56.1 MB/s \n",
            "\u001b[?25hCollecting py-spy>=0.2.0\n",
            "  Downloading py_spy-0.3.11-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0 MB 40.8 MB/s \n",
            "\u001b[?25hCollecting gpustat>=1.0.0b1\n",
            "  Downloading gpustat-1.0.0b1.tar.gz (82 kB)\n",
            "\u001b[K     |████████████████████████████████| 82 kB 241 kB/s \n",
            "\u001b[?25hCollecting aioredis<2\n",
            "  Downloading aioredis-1.3.1-py3-none-any.whl (65 kB)\n",
            "\u001b[K     |████████████████████████████████| 65 kB 3.7 MB/s \n",
            "\u001b[?25hCollecting aiohttp-cors\n",
            "  Downloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)\n",
            "Collecting hiredis\n",
            "  Downloading hiredis-2.0.0-cp37-cp37m-manylinux2010_x86_64.whl (85 kB)\n",
            "\u001b[K     |████████████████████████████████| 85 kB 3.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: nvidia-ml-py3>=7.352.0 in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[default]->finrl==0.3.4) (7.352.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[default]->finrl==0.3.4) (5.4.8)\n",
            "Collecting blessed>=1.17.1\n",
            "  Downloading blessed-1.19.0-py2.py3-none-any.whl (57 kB)\n",
            "\u001b[K     |████████████████████████████████| 57 kB 4.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.7/dist-packages (from redis>=3.5.0->ray[default]->finrl==0.3.4) (21.3)\n",
            "Collecting deprecated>=1.2.3\n",
            "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.7/dist-packages (from deprecated>=1.2.3->redis>=3.5.0->ray[default]->finrl==0.3.4) (1.13.3)\n",
            "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->ray[default]->finrl==0.3.4) (5.4.0)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->ray[default]->finrl==0.3.4) (0.18.0)\n",
            "Collecting opencensus-context==0.1.2\n",
            "  Downloading opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)\n",
            "Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from opencensus->ray[default]->finrl==0.3.4) (1.26.3)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (1.54.0)\n",
            "Requirement already satisfied: google-auth<2.0dev,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (1.35.0)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (4.2.4)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (4.8)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.4) (0.4.8)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.4) (0.8.9)\n",
            "Requirement already satisfied: atari-py~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.4) (0.2.9)\n",
            "Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.4) (2.7.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.4) (7.1.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (0.6.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (3.3.6)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (0.4.6)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (0.12.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (1.8.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (1.3.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.4) (3.1.1)\n",
            "Collecting psycopg2-binary\n",
            "  Downloading psycopg2_binary-2.9.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0 MB 52.7 MB/s \n",
            "\u001b[?25hCollecting mock\n",
            "  Downloading mock-4.0.3-py3-none-any.whl (28 kB)\n",
            "Collecting requests>=2.18.4\n",
            "  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)\n",
            "\u001b[K     |████████████████████████████████| 63 kB 1.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance->finrl==0.3.4) (0.0.10)\n",
            "Collecting lxml\n",
            "  Downloading lxml-4.7.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.4 MB 43.3 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: finrl, elegantrl, pyfolio, empyrical, exchange-calendars, gputil, thriftpy2, gpustat\n",
            "  Building wheel for finrl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for finrl: filename=finrl-0.3.4-py3-none-any.whl size=3885434 sha256=808522742e46cd84b13f58b4af43111a78b19bf31ed2d9fb72106bea9f5edee2\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-064dnzxq/wheels/17/ff/bd/1bc602a0352762b0b24041b88536d803ae343ed0a711fcf55e\n",
            "  Building wheel for elegantrl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for elegantrl: filename=elegantrl-0.3.3-py3-none-any.whl size=188472 sha256=9004c719344468911008f925ffc82d9a0f9f6709a98e91b1da569dc63b8841f9\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-064dnzxq/wheels/99/85/5e/86cb3a9f47adfca5e248295e93113e1b298d60883126d62c84\n",
            "  Building wheel for pyfolio (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyfolio: filename=pyfolio-0.9.2+75.g4b901f6-py3-none-any.whl size=75774 sha256=fe27a5ff34eca3ed3d6a2ccf78a5cb74d1c2b754af0ffd75e695c43bccf05e81\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-064dnzxq/wheels/ef/09/e5/2c1bf37c050d22557c080deb1be986d06424627c04aeca19b9\n",
            "  Building wheel for empyrical (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for empyrical: filename=empyrical-0.5.5-py3-none-any.whl size=39780 sha256=ea2cab39223178834a8758f2077c9ff3e90bbb15ec56371792aa4556607ff243\n",
            "  Stored in directory: /root/.cache/pip/wheels/d9/91/4b/654fcff57477efcf149eaca236da2fce991526cbab431bf312\n",
            "  Building wheel for exchange-calendars (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for exchange-calendars: filename=exchange_calendars-3.5-py3-none-any.whl size=179486 sha256=8853493f9576ef9cb55fff1951b2052d01e61b29508045638e3a2935df153fda\n",
            "  Stored in directory: /root/.cache/pip/wheels/69/21/43/b6ae2605dd767f6cd5a5b0b70c93a9a75823e44b3ccb92bce7\n",
            "  Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gputil: filename=GPUtil-1.4.0-py3-none-any.whl size=7411 sha256=a8cb8c440ca82f382fd0c4a4df57d1e466fae2a9413af5cd3792eb74fb0a0268\n",
            "  Stored in directory: /root/.cache/pip/wheels/6e/f8/83/534c52482d6da64622ddbf72cd93c35d2ef2881b78fd08ff0c\n",
            "  Building wheel for thriftpy2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for thriftpy2: filename=thriftpy2-0.4.14-cp37-cp37m-linux_x86_64.whl size=944210 sha256=93c31f8546b311a0abaacaa2983e9e6482747a332f45a0ef7ad1341f7daab8d1\n",
            "  Stored in directory: /root/.cache/pip/wheels/2a/f5/49/9c0d851aa64b58db72883cf9393cc824d536bdf13f5c83cff4\n",
            "  Building wheel for gpustat (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gpustat: filename=gpustat-1.0.0b1-py3-none-any.whl size=15979 sha256=56bf2758ebfe36a32232e8a66eea48f1b4a77c103de7ce5b97670c28ce701249\n",
            "  Stored in directory: /root/.cache/pip/wheels/1a/16/e2/3e2437fba4c4b6a97a97bd96fce5d14e66cff5c4966fb1cc8c\n",
            "Successfully built finrl elegantrl pyfolio empyrical exchange-calendars gputil thriftpy2 gpustat\n",
            "Installing collected packages: requests, multidict, frozenlist, yarl, lxml, deprecated, asynctest, async-timeout, aiosignal, redis, pyyaml, pycares, ply, platformdirs, opencensus-context, msgpack, hiredis, distlib, blessed, aiohttp, websockets, websocket-client, virtualenv, thriftpy2, tensorboardX, stable-baselines3, ray, pymysql, pyluach, pybullet, py-spy, psycopg2-binary, opencensus, nodeenv, mock, identify, gpustat, empyrical, cryptography, colorful, cfgv, box2d-py, aioredis, aiohttp-cors, aiodns, yfinance, wrds, stockstats, pyfolio, pre-commit, lz4, jqdatasdk, gputil, exchange-calendars, elegantrl, ccxt, alpaca-trade-api, finrl\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Attempting uninstall: lxml\n",
            "    Found existing installation: lxml 4.2.6\n",
            "    Uninstalling lxml-4.2.6:\n",
            "      Successfully uninstalled lxml-4.2.6\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Attempting uninstall: msgpack\n",
            "    Found existing installation: msgpack 1.0.3\n",
            "    Uninstalling msgpack-1.0.3:\n",
            "      Successfully uninstalled msgpack-1.0.3\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.27.1 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Successfully installed aiodns-3.0.0 aiohttp-3.8.1 aiohttp-cors-0.7.0 aioredis-1.3.1 aiosignal-1.2.0 alpaca-trade-api-1.2.3 async-timeout-4.0.2 asynctest-0.13.0 blessed-1.19.0 box2d-py-2.3.8 ccxt-1.67.31 cfgv-3.3.1 colorful-0.5.4 cryptography-36.0.1 deprecated-1.2.13 distlib-0.3.4 elegantrl-0.3.3 empyrical-0.5.5 exchange-calendars-3.5 finrl-0.3.4 frozenlist-1.2.0 gpustat-1.0.0b1 gputil-1.4.0 hiredis-2.0.0 identify-2.4.3 jqdatasdk-1.8.10 lxml-4.7.1 lz4-3.1.10 mock-4.0.3 msgpack-1.0.2 multidict-5.2.0 nodeenv-1.6.0 opencensus-0.8.0 opencensus-context-0.1.2 platformdirs-2.4.1 ply-3.11 pre-commit-2.16.0 psycopg2-binary-2.9.3 py-spy-0.3.11 pybullet-3.2.1 pycares-4.1.2 pyfolio-0.9.2+75.g4b901f6 pyluach-1.3.0 pymysql-1.0.2 pyyaml-6.0 ray-1.9.2 redis-4.1.0 requests-2.27.1 stable-baselines3-1.3.0 stockstats-0.4.1 tensorboardX-2.4.1 thriftpy2-0.4.14 virtualenv-20.13.0 websocket-client-1.2.3 websockets-9.1 wrds-3.1.1 yarl-1.7.2 yfinance-0.1.69\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h2568cp5bU38"
      },
      "source": [
        "\n",
        "<a id='1.2'></a>\n",
        "## 2.2. Check if the additional packages needed are present, if not install them. \n",
        "* Yahoo Finance API\n",
        "* pandas\n",
        "* numpy\n",
        "* matplotlib\n",
        "* stockstats\n",
        "* OpenAI gym\n",
        "* stable-baselines\n",
        "* tensorflow\n",
        "* pyfolio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bNmvYN9YbU4B"
      },
      "source": [
        "<a id='1.3'></a>\n",
        "## 2.3. Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ntfTb0e2bU4C",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2267190b-fe54-43a3-8d03-893d02a20f39",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "matplotlib.use('Agg')\n",
        "%matplotlib inline\n",
        "import datetime\n",
        "\n",
        "from finrl.apps import config\n",
        "from finrl.finrl_meta.preprocessor.yahoodownloader import YahooDownloader\n",
        "from finrl.finrl_meta.preprocessor.preprocessors import FeatureEngineer, data_split\n",
        "from finrl.finrl_meta.env_portfolio_allocation.env_portfolio import StockPortfolioEnv\n",
        "from finrl.drl_agents.stablebaselines3.models import DRLAgent\n",
        "from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline,convert_daily_return_to_pyfolio_ts\n",
        "from finrl.finrl_meta.data_processor import DataProcessor\n",
        "from finrl.finrl_meta.data_processors.processor_yahoofinance import YahooFinanceProcessor\n",
        "import sys\n",
        "sys.path.append(\"../FinRL-Library\")"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pyfolio/pos.py:27: UserWarning: Module \"zipline.assets\" not found; multipliers will not be applied to position notionals.\n",
            "  'Module \"zipline.assets\" not found; multipliers will not be applied'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlIS2abxkwan"
      },
      "source": [
        "<a id='1.4'></a>\n",
        "## 2.4. Create Folders"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B8bBq7nsBCfF",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "import os\n",
        "if not os.path.exists(\"./\" + config.DATA_SAVE_DIR):\n",
        "    os.makedirs(\"./\" + config.DATA_SAVE_DIR)\n",
        "if not os.path.exists(\"./\" + config.TRAINED_MODEL_DIR):\n",
        "    os.makedirs(\"./\" + config.TRAINED_MODEL_DIR)\n",
        "if not os.path.exists(\"./\" + config.TENSORBOARD_LOG_DIR):\n",
        "    os.makedirs(\"./\" + config.TENSORBOARD_LOG_DIR)\n",
        "if not os.path.exists(\"./\" + config.RESULTS_DIR):\n",
        "    os.makedirs(\"./\" + config.RESULTS_DIR)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slBria_QbU4F"
      },
      "source": [
        "<a id='2'></a>\n",
        "# Part 3. Download Data\n",
        "Yahoo Finance is a website that provides stock data, financial news, financial reports, etc. All the data provided by Yahoo Finance is free.\n",
        "* FinRL uses a class **YahooDownloader** to fetch data from Yahoo Finance API\n",
        "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CPsuy6d9yRPp",
        "outputId": "5f9be2b1-c3f9-4205-a753-a81a1d793d17",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "print(config.DOW_30_TICKER)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['AXP', 'AMGN', 'AAPL', 'BA', 'CAT', 'CSCO', 'CVX', 'GS', 'HD', 'HON', 'IBM', 'INTC', 'JNJ', 'KO', 'JPM', 'MCD', 'MMM', 'MRK', 'MSFT', 'NKE', 'PG', 'TRV', 'UNH', 'CRM', 'VZ', 'V', 'WBA', 'WMT', 'DIS', 'DOW']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WEwzMkFHbU4G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8db19feb-cf14-418f-d6d4-99aeb0bfa6ef",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "dp = YahooFinanceProcessor()\n",
        "df = dp.download_data(start_date = '2008-01-01',\n",
        "                     end_date = '2021-10-31',\n",
        "                     ticker_list = config.DOW_30_TICKER, time_interval='1D')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "Shape of DataFrame:  (101615, 9)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V1xC-LpbbU4f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "d601ae03-2909-487e-b5f5-2db9d4e3f8cb",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-2c7f7c38-fd3a-4c70-a06e-2005a497e380\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>adjcp</th>\n",
              "      <th>volume</th>\n",
              "      <th>tic</th>\n",
              "      <th>day</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>7.116786</td>\n",
              "      <td>7.152143</td>\n",
              "      <td>6.876786</td>\n",
              "      <td>6.958571</td>\n",
              "      <td>5.966037</td>\n",
              "      <td>1079178800</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>46.599998</td>\n",
              "      <td>47.040001</td>\n",
              "      <td>46.259998</td>\n",
              "      <td>46.599998</td>\n",
              "      <td>36.034901</td>\n",
              "      <td>7934400</td>\n",
              "      <td>AMGN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>52.090000</td>\n",
              "      <td>52.320000</td>\n",
              "      <td>50.790001</td>\n",
              "      <td>51.040001</td>\n",
              "      <td>40.592686</td>\n",
              "      <td>8053700</td>\n",
              "      <td>AXP</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>87.570000</td>\n",
              "      <td>87.839996</td>\n",
              "      <td>86.000000</td>\n",
              "      <td>86.620003</td>\n",
              "      <td>63.481625</td>\n",
              "      <td>4303000</td>\n",
              "      <td>BA</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>72.559998</td>\n",
              "      <td>72.669998</td>\n",
              "      <td>70.050003</td>\n",
              "      <td>70.629997</td>\n",
              "      <td>47.633076</td>\n",
              "      <td>6337800</td>\n",
              "      <td>CAT</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2c7f7c38-fd3a-4c70-a06e-2005a497e380')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2c7f7c38-fd3a-4c70-a06e-2005a497e380 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2c7f7c38-fd3a-4c70-a06e-2005a497e380');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "         date       open       high  ...      volume   tic  day\n",
              "0  2008-01-02   7.116786   7.152143  ...  1079178800  AAPL    2\n",
              "1  2008-01-02  46.599998  47.040001  ...     7934400  AMGN    2\n",
              "2  2008-01-02  52.090000  52.320000  ...     8053700   AXP    2\n",
              "3  2008-01-02  87.570000  87.839996  ...     4303000    BA    2\n",
              "4  2008-01-02  72.559998  72.669998  ...     6337800   CAT    2\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mS1-nxRzbU4i",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6ba62b6c-f24d-453b-8fc5-4bf1d58d0983",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(101615, 9)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V9UwKwzRbU4l"
      },
      "source": [
        "# Part 4: Preprocess Data\n",
        "Data preprocessing is a crucial step for training a high quality machine learning model. We need to check for missing data and do feature engineering in order to convert the data into a model-ready state.\n",
        "* Add technical indicators. In practical trading, various information needs to be taken into account, for example the historical stock prices, current holding shares, technical indicators, etc. In this article, we demonstrate two trend-following technical indicators: MACD and RSI.\n",
        "* Add turbulence index. Risk-aversion reflects whether an investor will choose to preserve the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the financial turbulence index that measures extreme asset price fluctuation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P5h8RbeBHMDQ",
        "outputId": "dfa222bf-1363-4354-f825-28d86f50bbff",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "fe = FeatureEngineer(\n",
        "                    use_technical_indicator=True,\n",
        "                    use_turbulence=False,\n",
        "                    user_defined_feature = False)\n",
        "\n",
        "df = fe.preprocess_data(df)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully added technical indicators\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_zsIW1LAiqCb",
        "outputId": "080be760-c4b2-4115-db37-b132cd482971",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(97524, 17)"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lWB-RoTSbU4s",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "3177f1b3-46e9-4310-9567-280527943c47",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-834d83d0-fa2e-4ae0-9af4-eb6ce00a3a1c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>adjcp</th>\n",
              "      <th>volume</th>\n",
              "      <th>tic</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>7.116786</td>\n",
              "      <td>7.152143</td>\n",
              "      <td>6.876786</td>\n",
              "      <td>6.958571</td>\n",
              "      <td>5.966037</td>\n",
              "      <td>1079178800</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.964725</td>\n",
              "      <td>6.955632</td>\n",
              "      <td>100.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>100.0</td>\n",
              "      <td>6.958571</td>\n",
              "      <td>6.958571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3483</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>46.599998</td>\n",
              "      <td>47.040001</td>\n",
              "      <td>46.259998</td>\n",
              "      <td>46.599998</td>\n",
              "      <td>36.034901</td>\n",
              "      <td>7934400</td>\n",
              "      <td>AMGN</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.964725</td>\n",
              "      <td>6.955632</td>\n",
              "      <td>100.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>100.0</td>\n",
              "      <td>46.599998</td>\n",
              "      <td>46.599998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6966</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>52.090000</td>\n",
              "      <td>52.320000</td>\n",
              "      <td>50.790001</td>\n",
              "      <td>51.040001</td>\n",
              "      <td>40.592686</td>\n",
              "      <td>8053700</td>\n",
              "      <td>AXP</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.964725</td>\n",
              "      <td>6.955632</td>\n",
              "      <td>100.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>100.0</td>\n",
              "      <td>51.040001</td>\n",
              "      <td>51.040001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10449</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>87.570000</td>\n",
              "      <td>87.839996</td>\n",
              "      <td>86.000000</td>\n",
              "      <td>86.620003</td>\n",
              "      <td>63.481625</td>\n",
              "      <td>4303000</td>\n",
              "      <td>BA</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.964725</td>\n",
              "      <td>6.955632</td>\n",
              "      <td>100.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>100.0</td>\n",
              "      <td>86.620003</td>\n",
              "      <td>86.620003</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13932</th>\n",
              "      <td>2008-01-02</td>\n",
              "      <td>72.559998</td>\n",
              "      <td>72.669998</td>\n",
              "      <td>70.050003</td>\n",
              "      <td>70.629997</td>\n",
              "      <td>47.633076</td>\n",
              "      <td>6337800</td>\n",
              "      <td>CAT</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6.964725</td>\n",
              "      <td>6.955632</td>\n",
              "      <td>100.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>100.0</td>\n",
              "      <td>70.629997</td>\n",
              "      <td>70.629997</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-834d83d0-fa2e-4ae0-9af4-eb6ce00a3a1c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-834d83d0-fa2e-4ae0-9af4-eb6ce00a3a1c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-834d83d0-fa2e-4ae0-9af4-eb6ce00a3a1c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "             date       open       high  ...  dx_30  close_30_sma  close_60_sma\n",
              "0      2008-01-02   7.116786   7.152143  ...  100.0      6.958571      6.958571\n",
              "3483   2008-01-02  46.599998  47.040001  ...  100.0     46.599998     46.599998\n",
              "6966   2008-01-02  52.090000  52.320000  ...  100.0     51.040001     51.040001\n",
              "10449  2008-01-02  87.570000  87.839996  ...  100.0     86.620003     86.620003\n",
              "13932  2008-01-02  72.559998  72.669998  ...  100.0     70.629997     70.629997\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qz9K2vul6RmK"
      },
      "source": [
        "## Add covariance matrix as states"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IhizvNwcrg1n",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "# add covariance matrix as states\n",
        "df=df.sort_values(['date','tic'],ignore_index=True)\n",
        "df.index = df.date.factorize()[0]\n",
        "\n",
        "cov_list = []\n",
        "return_list = []\n",
        "\n",
        "# look back is one year\n",
        "lookback=252\n",
        "for i in range(lookback,len(df.index.unique())):\n",
        "  data_lookback = df.loc[i-lookback:i,:]\n",
        "  price_lookback=data_lookback.pivot_table(index = 'date',columns = 'tic', values = 'close')\n",
        "  return_lookback = price_lookback.pct_change().dropna()\n",
        "  return_list.append(return_lookback)\n",
        "\n",
        "  covs = return_lookback.cov().values \n",
        "  cov_list.append(covs)\n",
        "\n",
        "  \n",
        "df_cov = pd.DataFrame({'date':df.date.unique()[lookback:],'cov_list':cov_list,'return_list':return_list})\n",
        "df = df.merge(df_cov, on='date')\n",
        "df = df.sort_values(['date','tic']).reset_index(drop=True)\n",
        "        "
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dUPwwa13uBQ-",
        "outputId": "2aa538ba-9935-4f69-b8c8-2cf3a3a55e42",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(90468, 19)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "id": "wv3jR1zPrg4g",
        "outputId": "94cef7cc-830b-4c81-da8b-9efba5ad989a",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-fcf66a75-c1f3-4a78-a0dc-7d0d80123fab\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>adjcp</th>\n",
              "      <th>volume</th>\n",
              "      <th>tic</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "      <th>cov_list</th>\n",
              "      <th>return_list</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2008-12-31</td>\n",
              "      <td>3.070357</td>\n",
              "      <td>3.133571</td>\n",
              "      <td>3.047857</td>\n",
              "      <td>3.048214</td>\n",
              "      <td>2.613432</td>\n",
              "      <td>607541200</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>2</td>\n",
              "      <td>-0.097446</td>\n",
              "      <td>3.649552</td>\n",
              "      <td>2.895305</td>\n",
              "      <td>42.254771</td>\n",
              "      <td>-80.847207</td>\n",
              "      <td>16.129793</td>\n",
              "      <td>3.243631</td>\n",
              "      <td>3.375887</td>\n",
              "      <td>[[0.001348968986171653, 0.00042841264280825875...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2008-12-31</td>\n",
              "      <td>57.110001</td>\n",
              "      <td>58.220001</td>\n",
              "      <td>57.060001</td>\n",
              "      <td>57.750000</td>\n",
              "      <td>44.657001</td>\n",
              "      <td>6287200</td>\n",
              "      <td>AMGN</td>\n",
              "      <td>2</td>\n",
              "      <td>0.216368</td>\n",
              "      <td>58.947401</td>\n",
              "      <td>56.388599</td>\n",
              "      <td>51.060614</td>\n",
              "      <td>51.895357</td>\n",
              "      <td>10.432018</td>\n",
              "      <td>56.671334</td>\n",
              "      <td>56.044333</td>\n",
              "      <td>[[0.001348968986171653, 0.00042841264280825875...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2008-12-31</td>\n",
              "      <td>17.969999</td>\n",
              "      <td>18.750000</td>\n",
              "      <td>17.910000</td>\n",
              "      <td>18.549999</td>\n",
              "      <td>14.950781</td>\n",
              "      <td>9625600</td>\n",
              "      <td>AXP</td>\n",
              "      <td>2</td>\n",
              "      <td>-1.191668</td>\n",
              "      <td>23.723023</td>\n",
              "      <td>16.106977</td>\n",
              "      <td>42.521170</td>\n",
              "      <td>-74.811722</td>\n",
              "      <td>25.776759</td>\n",
              "      <td>20.030000</td>\n",
              "      <td>22.412000</td>\n",
              "      <td>[[0.001348968986171653, 0.00042841264280825875...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2008-12-31</td>\n",
              "      <td>41.590000</td>\n",
              "      <td>43.049999</td>\n",
              "      <td>41.500000</td>\n",
              "      <td>42.669998</td>\n",
              "      <td>32.005890</td>\n",
              "      <td>5443100</td>\n",
              "      <td>BA</td>\n",
              "      <td>2</td>\n",
              "      <td>-0.391219</td>\n",
              "      <td>42.894634</td>\n",
              "      <td>38.486366</td>\n",
              "      <td>47.290375</td>\n",
              "      <td>157.922391</td>\n",
              "      <td>5.366299</td>\n",
              "      <td>40.432000</td>\n",
              "      <td>43.304500</td>\n",
              "      <td>[[0.001348968986171653, 0.00042841264280825875...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2008-12-31</td>\n",
              "      <td>43.700001</td>\n",
              "      <td>45.099998</td>\n",
              "      <td>43.700001</td>\n",
              "      <td>44.669998</td>\n",
              "      <td>30.925035</td>\n",
              "      <td>6277400</td>\n",
              "      <td>CAT</td>\n",
              "      <td>2</td>\n",
              "      <td>0.979845</td>\n",
              "      <td>45.785565</td>\n",
              "      <td>38.404435</td>\n",
              "      <td>51.073052</td>\n",
              "      <td>98.904653</td>\n",
              "      <td>26.331746</td>\n",
              "      <td>40.266000</td>\n",
              "      <td>39.918333</td>\n",
              "      <td>[[0.001348968986171653, 0.00042841264280825875...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fcf66a75-c1f3-4a78-a0dc-7d0d80123fab')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-fcf66a75-c1f3-4a78-a0dc-7d0d80123fab button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-fcf66a75-c1f3-4a78-a0dc-7d0d80123fab');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "         date  ...                                        return_list\n",
              "0  2008-12-31  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "1  2008-12-31  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "2  2008-12-31  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "3  2008-12-31  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "4  2008-12-31  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "\n",
              "[5 rows x 19 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UooHj1OgbU4v"
      },
      "source": [
        "<a id='4'></a>\n",
        "# Part 5. Design Environment\n",
        "Considering the stochastic and interactive nature of the automated stock trading tasks, a financial task is modeled as a **Markov Decision Process (MDP)** problem. The training process involves observing stock price change, taking an action and reward's calculation to have the agent adjusting its strategy accordingly. By interacting with the environment, the trading agent will derive a trading strategy with the maximized rewards as time proceeds.\n",
        "\n",
        "Our trading environments, based on OpenAI Gym framework, simulate live stock markets with real market data according to the principle of time-driven simulation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQnmN1qdk88I"
      },
      "source": [
        "## Training data split: 2009-01-01 to 2020-07-01"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NrPxgv4eBQ_R",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "train = data_split(df, '2009-01-01','2020-07-01')\n",
        "#trade = data_split(df, '2020-01-01', config.END_DATE)"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "id": "vU2vXEll0hfk",
        "outputId": "e8c76dfd-2c43-4f60-8eee-a1a4f564eed6",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "train.head()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-1c08030e-808f-44e7-b2ae-5d739276d588\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>adjcp</th>\n",
              "      <th>volume</th>\n",
              "      <th>tic</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "      <th>cov_list</th>\n",
              "      <th>return_list</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>3.067143</td>\n",
              "      <td>3.251429</td>\n",
              "      <td>3.041429</td>\n",
              "      <td>3.241071</td>\n",
              "      <td>2.778781</td>\n",
              "      <td>746015200</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>4</td>\n",
              "      <td>-0.082758</td>\n",
              "      <td>3.633600</td>\n",
              "      <td>2.892864</td>\n",
              "      <td>45.440193</td>\n",
              "      <td>-30.508777</td>\n",
              "      <td>2.140064</td>\n",
              "      <td>3.244631</td>\n",
              "      <td>3.376833</td>\n",
              "      <td>[[0.001366150662406761, 0.000433938195725591, ...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>58.590000</td>\n",
              "      <td>59.080002</td>\n",
              "      <td>57.750000</td>\n",
              "      <td>58.990002</td>\n",
              "      <td>45.615871</td>\n",
              "      <td>6547900</td>\n",
              "      <td>AMGN</td>\n",
              "      <td>4</td>\n",
              "      <td>0.320448</td>\n",
              "      <td>59.148360</td>\n",
              "      <td>56.339640</td>\n",
              "      <td>52.756859</td>\n",
              "      <td>94.549630</td>\n",
              "      <td>0.814217</td>\n",
              "      <td>56.759667</td>\n",
              "      <td>56.166000</td>\n",
              "      <td>[[0.001366150662406761, 0.000433938195725591, ...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>18.570000</td>\n",
              "      <td>19.520000</td>\n",
              "      <td>18.400000</td>\n",
              "      <td>19.330000</td>\n",
              "      <td>15.579438</td>\n",
              "      <td>10955700</td>\n",
              "      <td>AXP</td>\n",
              "      <td>4</td>\n",
              "      <td>-1.059847</td>\n",
              "      <td>23.489423</td>\n",
              "      <td>16.086577</td>\n",
              "      <td>43.923322</td>\n",
              "      <td>-42.018825</td>\n",
              "      <td>16.335101</td>\n",
              "      <td>20.028333</td>\n",
              "      <td>22.263333</td>\n",
              "      <td>[[0.001366150662406761, 0.000433938195725591, ...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>42.799999</td>\n",
              "      <td>45.560001</td>\n",
              "      <td>42.779999</td>\n",
              "      <td>45.250000</td>\n",
              "      <td>33.941097</td>\n",
              "      <td>7010200</td>\n",
              "      <td>BA</td>\n",
              "      <td>4</td>\n",
              "      <td>-0.019566</td>\n",
              "      <td>43.926849</td>\n",
              "      <td>37.932151</td>\n",
              "      <td>50.664690</td>\n",
              "      <td>275.696308</td>\n",
              "      <td>20.494464</td>\n",
              "      <td>40.621667</td>\n",
              "      <td>43.237334</td>\n",
              "      <td>[[0.001366150662406761, 0.000433938195725591, ...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>44.910000</td>\n",
              "      <td>46.980000</td>\n",
              "      <td>44.709999</td>\n",
              "      <td>46.910000</td>\n",
              "      <td>32.475800</td>\n",
              "      <td>7117200</td>\n",
              "      <td>CAT</td>\n",
              "      <td>4</td>\n",
              "      <td>1.248426</td>\n",
              "      <td>46.543072</td>\n",
              "      <td>38.372928</td>\n",
              "      <td>53.534743</td>\n",
              "      <td>131.675975</td>\n",
              "      <td>34.637448</td>\n",
              "      <td>40.623333</td>\n",
              "      <td>39.911333</td>\n",
              "      <td>[[0.001366150662406761, 0.000433938195725591, ...</td>\n",
              "      <td>tic             AAPL      AMGN       AXP  ... ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1c08030e-808f-44e7-b2ae-5d739276d588')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1c08030e-808f-44e7-b2ae-5d739276d588 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1c08030e-808f-44e7-b2ae-5d739276d588');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "         date  ...                                        return_list\n",
              "0  2009-01-02  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "0  2009-01-02  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "0  2009-01-02  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "0  2009-01-02  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "0  2009-01-02  ...  tic             AAPL      AMGN       AXP  ... ...\n",
              "\n",
              "[5 rows x 19 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxQTNjpblAMN"
      },
      "source": [
        "## Environment for Portfolio Allocation\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xlfE-VERbU40",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from gym.utils import seeding\n",
        "import gym\n",
        "from gym import spaces\n",
        "import matplotlib\n",
        "matplotlib.use('Agg')\n",
        "import matplotlib.pyplot as plt\n",
        "from stable_baselines3.common.vec_env import DummyVecEnv\n",
        "\n",
        "\n",
        "class StockPortfolioEnv(gym.Env):\n",
        "    \"\"\"A single stock trading environment for OpenAI gym\n",
        "\n",
        "    Attributes\n",
        "    ----------\n",
        "        df: DataFrame\n",
        "            input data\n",
        "        stock_dim : int\n",
        "            number of unique stocks\n",
        "        hmax : int\n",
        "            maximum number of shares to trade\n",
        "        initial_amount : int\n",
        "            start money\n",
        "        transaction_cost_pct: float\n",
        "            transaction cost percentage per trade\n",
        "        reward_scaling: float\n",
        "            scaling factor for reward, good for training\n",
        "        state_space: int\n",
        "            the dimension of input features\n",
        "        action_space: int\n",
        "            equals stock dimension\n",
        "        tech_indicator_list: list\n",
        "            a list of technical indicator names\n",
        "        turbulence_threshold: int\n",
        "            a threshold to control risk aversion\n",
        "        day: int\n",
        "            an increment number to control date\n",
        "\n",
        "    Methods\n",
        "    -------\n",
        "    _sell_stock()\n",
        "        perform sell action based on the sign of the action\n",
        "    _buy_stock()\n",
        "        perform buy action based on the sign of the action\n",
        "    step()\n",
        "        at each step the agent will return actions, then \n",
        "        we will calculate the reward, and return the next observation.\n",
        "    reset()\n",
        "        reset the environment\n",
        "    render()\n",
        "        use render to return other functions\n",
        "    save_asset_memory()\n",
        "        return account value at each time step\n",
        "    save_action_memory()\n",
        "        return actions/positions at each time step\n",
        "        \n",
        "\n",
        "    \"\"\"\n",
        "    metadata = {'render.modes': ['human']}\n",
        "\n",
        "    def __init__(self, \n",
        "                df,\n",
        "                stock_dim,\n",
        "                hmax,\n",
        "                initial_amount,\n",
        "                transaction_cost_pct,\n",
        "                reward_scaling,\n",
        "                state_space,\n",
        "                action_space,\n",
        "                tech_indicator_list,\n",
        "                turbulence_threshold=None,\n",
        "                lookback=252,\n",
        "                day = 0):\n",
        "        #super(StockEnv, self).__init__()\n",
        "        #money = 10 , scope = 1\n",
        "        self.day = day\n",
        "        self.lookback=lookback\n",
        "        self.df = df\n",
        "        self.stock_dim = stock_dim\n",
        "        self.hmax = hmax\n",
        "        self.initial_amount = initial_amount\n",
        "        self.transaction_cost_pct =transaction_cost_pct\n",
        "        self.reward_scaling = reward_scaling\n",
        "        self.state_space = state_space\n",
        "        self.action_space = action_space\n",
        "        self.tech_indicator_list = tech_indicator_list\n",
        "\n",
        "        # action_space normalization and shape is self.stock_dim\n",
        "        self.action_space = spaces.Box(low = 0, high = 1,shape = (self.action_space,)) \n",
        "        # Shape = (34, 30)\n",
        "        # covariance matrix + technical indicators\n",
        "        self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape = (self.state_space+len(self.tech_indicator_list),self.state_space))\n",
        "\n",
        "        # load data from a pandas dataframe\n",
        "        self.data = self.df.loc[self.day,:]\n",
        "        self.covs = self.data['cov_list'].values[0]\n",
        "        self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)\n",
        "        self.terminal = False     \n",
        "        self.turbulence_threshold = turbulence_threshold        \n",
        "        # initalize state: inital portfolio return + individual stock return + individual weights\n",
        "        self.portfolio_value = self.initial_amount\n",
        "\n",
        "        # memorize portfolio value each step\n",
        "        self.asset_memory = [self.initial_amount]\n",
        "        # memorize portfolio return each step\n",
        "        self.portfolio_return_memory = [0]\n",
        "        self.actions_memory=[[1/self.stock_dim]*self.stock_dim]\n",
        "        self.date_memory=[self.data.date.unique()[0]]\n",
        "\n",
        "        \n",
        "    def step(self, actions):\n",
        "        # print(self.day)\n",
        "        self.terminal = self.day >= len(self.df.index.unique())-1\n",
        "        # print(actions)\n",
        "\n",
        "        if self.terminal:\n",
        "            df = pd.DataFrame(self.portfolio_return_memory)\n",
        "            df.columns = ['daily_return']\n",
        "            plt.plot(df.daily_return.cumsum(),'r')\n",
        "            plt.savefig('results/cumulative_reward.png')\n",
        "            plt.close()\n",
        "            \n",
        "            plt.plot(self.portfolio_return_memory,'r')\n",
        "            plt.savefig('results/rewards.png')\n",
        "            plt.close()\n",
        "\n",
        "            print(\"=================================\")\n",
        "            print(\"begin_total_asset:{}\".format(self.asset_memory[0]))           \n",
        "            print(\"end_total_asset:{}\".format(self.portfolio_value))\n",
        "\n",
        "            df_daily_return = pd.DataFrame(self.portfolio_return_memory)\n",
        "            df_daily_return.columns = ['daily_return']\n",
        "            if df_daily_return['daily_return'].std() !=0:\n",
        "              sharpe = (252**0.5)*df_daily_return['daily_return'].mean()/ \\\n",
        "                       df_daily_return['daily_return'].std()\n",
        "              print(\"Sharpe: \",sharpe)\n",
        "            print(\"=================================\")\n",
        "            \n",
        "            return self.state, self.reward, self.terminal,{}\n",
        "\n",
        "        else:\n",
        "            #print(\"Model actions: \",actions)\n",
        "            # actions are the portfolio weight\n",
        "            # normalize to sum of 1\n",
        "            #if (np.array(actions) - np.array(actions).min()).sum() != 0:\n",
        "            #  norm_actions = (np.array(actions) - np.array(actions).min()) / (np.array(actions) - np.array(actions).min()).sum()\n",
        "            #else:\n",
        "            #  norm_actions = actions\n",
        "            weights = self.softmax_normalization(actions) \n",
        "            #print(\"Normalized actions: \", weights)\n",
        "            self.actions_memory.append(weights)\n",
        "            last_day_memory = self.data\n",
        "\n",
        "            #load next state\n",
        "            self.day += 1\n",
        "            self.data = self.df.loc[self.day,:]\n",
        "            self.covs = self.data['cov_list'].values[0]\n",
        "            self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)\n",
        "            #print(self.state)\n",
        "            # calcualte portfolio return\n",
        "            # individual stocks' return * weight\n",
        "            portfolio_return = sum(((self.data.close.values / last_day_memory.close.values)-1)*weights)\n",
        "            # update portfolio value\n",
        "            new_portfolio_value = self.portfolio_value*(1+portfolio_return)\n",
        "            self.portfolio_value = new_portfolio_value\n",
        "\n",
        "            # save into memory\n",
        "            self.portfolio_return_memory.append(portfolio_return)\n",
        "            self.date_memory.append(self.data.date.unique()[0])            \n",
        "            self.asset_memory.append(new_portfolio_value)\n",
        "\n",
        "            # the reward is the new portfolio value or end portfolo value\n",
        "            self.reward = new_portfolio_value \n",
        "            #print(\"Step reward: \", self.reward)\n",
        "            #self.reward = self.reward*self.reward_scaling\n",
        "\n",
        "        return self.state, self.reward, self.terminal, {}\n",
        "\n",
        "    def reset(self):\n",
        "        self.asset_memory = [self.initial_amount]\n",
        "        self.day = 0\n",
        "        self.data = self.df.loc[self.day,:]\n",
        "        # load states\n",
        "        self.covs = self.data['cov_list'].values[0]\n",
        "        self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)\n",
        "        self.portfolio_value = self.initial_amount\n",
        "        #self.cost = 0\n",
        "        #self.trades = 0\n",
        "        self.terminal = False \n",
        "        self.portfolio_return_memory = [0]\n",
        "        self.actions_memory=[[1/self.stock_dim]*self.stock_dim]\n",
        "        self.date_memory=[self.data.date.unique()[0]] \n",
        "        return self.state\n",
        "    \n",
        "    def render(self, mode='human'):\n",
        "        return self.state\n",
        "        \n",
        "    def softmax_normalization(self, actions):\n",
        "        numerator = np.exp(actions)\n",
        "        denominator = np.sum(np.exp(actions))\n",
        "        softmax_output = numerator/denominator\n",
        "        return softmax_output\n",
        "\n",
        "    \n",
        "    def save_asset_memory(self):\n",
        "        date_list = self.date_memory\n",
        "        portfolio_return = self.portfolio_return_memory\n",
        "        #print(len(date_list))\n",
        "        #print(len(asset_list))\n",
        "        df_account_value = pd.DataFrame({'date':date_list,'daily_return':portfolio_return})\n",
        "        return df_account_value\n",
        "\n",
        "    def save_action_memory(self):\n",
        "        # date and close price length must match actions length\n",
        "        date_list = self.date_memory\n",
        "        df_date = pd.DataFrame(date_list)\n",
        "        df_date.columns = ['date']\n",
        "        \n",
        "        action_list = self.actions_memory\n",
        "        df_actions = pd.DataFrame(action_list)\n",
        "        df_actions.columns = self.data.tic.values\n",
        "        df_actions.index = df_date.date\n",
        "        #df_actions = pd.DataFrame({'date':date_list,'actions':action_list})\n",
        "        return df_actions\n",
        "\n",
        "    def _seed(self, seed=None):\n",
        "        self.np_random, seed = seeding.np_random(seed)\n",
        "        return [seed]\n",
        "\n",
        "    def get_sb_env(self):\n",
        "        e = DummyVecEnv([lambda: self])\n",
        "        obs = e.reset()\n",
        "        return e, obs"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MzD06X0CbU43",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab17eecd-2079-4684-a793-0541088b1c00",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "stock_dimension = len(train.tic.unique())\n",
        "state_space = stock_dimension\n",
        "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Stock Dimension: 28, State Space: 28\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jyg0_ZuVEVQ5",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "env_kwargs = {\n",
        "    \"hmax\": 100, \n",
        "    \"initial_amount\": 1000000, \n",
        "    \"transaction_cost_pct\": 0.001, \n",
        "    \"state_space\": state_space, \n",
        "    \"stock_dim\": stock_dimension, \n",
        "    \"tech_indicator_list\": config.TECHNICAL_INDICATORS_LIST, \n",
        "    \"action_space\": stock_dimension, \n",
        "    \"reward_scaling\": 1e-4\n",
        "    \n",
        "}\n",
        "\n",
        "e_train_gym = StockPortfolioEnv(df = train, **env_kwargs)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HTlOf8SJGdkl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1141154d-5cdc-41ba-f485-4d6e93680049",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "env_train, _ = e_train_gym.get_sb_env()\n",
        "print(type(env_train))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'stable_baselines3.common.vec_env.dummy_vec_env.DummyVecEnv'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2eKIu5UPlPnk"
      },
      "source": [
        "<a id='5'></a>\n",
        "# Part 6: Implement DRL Algorithms\n",
        "* The implementation of the DRL algorithms are based on **OpenAI Baselines** and **Stable Baselines**. Stable Baselines is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups.\n",
        "* FinRL library includes fine-tuned standard DRL algorithms, such as DQN, DDPG,\n",
        "Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
        "design their own DRL algorithms by adapting these DRL algorithms."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VDxU0iCEGdnb",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "# initialize\n",
        "agent = DRLAgent(env = env_train)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hdPe8uzflbXe"
      },
      "source": [
        "### Model 1: **A2C**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t1tORf1fIcQ2",
        "outputId": "4a1b885a-ef5d-4dff-f24f-c9468347067b",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "\n",
        "A2C_PARAMS = {\"n_steps\": 5, \"ent_coef\": 0.005, \"learning_rate\": 0.0002}\n",
        "model_a2c = agent.get_model(model_name=\"a2c\",model_kwargs = A2C_PARAMS)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0002}\n",
            "Using cpu device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DazEdrMpIdyz",
        "outputId": "540c591c-e0da-43f8-ae08-989d427a8c79",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_a2c = agent.train_model(model=model_a2c, \n",
        "                                tb_log_name='a2c',\n",
        "                                total_timesteps=50000)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 271       |\n",
            "|    iterations         | 100       |\n",
            "|    time_elapsed       | 1         |\n",
            "|    total_timesteps    | 500       |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 99        |\n",
            "|    policy_loss        | 1.73e+08  |\n",
            "|    reward             | 1567781.2 |\n",
            "|    std                | 0.996     |\n",
            "|    value_loss         | 2.59e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 200       |\n",
            "|    time_elapsed       | 3         |\n",
            "|    total_timesteps    | 1000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.6     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 199       |\n",
            "|    policy_loss        | 2.31e+08  |\n",
            "|    reward             | 1893147.8 |\n",
            "|    std                | 0.995     |\n",
            "|    value_loss         | 3.8e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 280       |\n",
            "|    iterations         | 300       |\n",
            "|    time_elapsed       | 5         |\n",
            "|    total_timesteps    | 1500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 299       |\n",
            "|    policy_loss        | 3.13e+08  |\n",
            "|    reward             | 2748750.5 |\n",
            "|    std                | 0.994     |\n",
            "|    value_loss         | 8.45e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 400       |\n",
            "|    time_elapsed       | 7         |\n",
            "|    total_timesteps    | 2000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 399       |\n",
            "|    policy_loss        | 3.58e+08  |\n",
            "|    reward             | 3250711.8 |\n",
            "|    std                | 0.994     |\n",
            "|    value_loss         | 1.06e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 280       |\n",
            "|    iterations         | 500       |\n",
            "|    time_elapsed       | 8         |\n",
            "|    total_timesteps    | 2500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 499       |\n",
            "|    policy_loss        | 5.11e+08  |\n",
            "|    reward             | 4155249.2 |\n",
            "|    std                | 0.993     |\n",
            "|    value_loss         | 1.92e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4443599.458563017\n",
            "Sharpe:  0.799061278872168\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 600       |\n",
            "|    time_elapsed       | 10        |\n",
            "|    total_timesteps    | 3000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 599       |\n",
            "|    policy_loss        | 1.3e+08   |\n",
            "|    reward             | 1079644.1 |\n",
            "|    std                | 0.993     |\n",
            "|    value_loss         | 1.16e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 700       |\n",
            "|    time_elapsed       | 12        |\n",
            "|    total_timesteps    | 3500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 699       |\n",
            "|    policy_loss        | 1.91e+08  |\n",
            "|    reward             | 1611516.5 |\n",
            "|    std                | 0.993     |\n",
            "|    value_loss         | 2.85e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 800       |\n",
            "|    time_elapsed       | 14        |\n",
            "|    total_timesteps    | 4000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 799       |\n",
            "|    policy_loss        | 2.54e+08  |\n",
            "|    reward             | 2210773.5 |\n",
            "|    std                | 0.992     |\n",
            "|    value_loss         | 5.25e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 900       |\n",
            "|    time_elapsed       | 16        |\n",
            "|    total_timesteps    | 4500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 899       |\n",
            "|    policy_loss        | 3.35e+08  |\n",
            "|    reward             | 2885866.2 |\n",
            "|    std                | 0.992     |\n",
            "|    value_loss         | 8.77e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 279       |\n",
            "|    iterations         | 1000      |\n",
            "|    time_elapsed       | 17        |\n",
            "|    total_timesteps    | 5000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | -2.38e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 999       |\n",
            "|    policy_loss        | 3.65e+08  |\n",
            "|    reward             | 3163569.8 |\n",
            "|    std                | 0.991     |\n",
            "|    value_loss         | 1.11e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 280       |\n",
            "|    iterations         | 1100      |\n",
            "|    time_elapsed       | 19        |\n",
            "|    total_timesteps    | 5500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1099      |\n",
            "|    policy_loss        | 5.32e+08  |\n",
            "|    reward             | 4098778.0 |\n",
            "|    std                | 0.991     |\n",
            "|    value_loss         | 1.9e+14   |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4197705.669681579\n",
            "Sharpe:  0.7761462637376377\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 1200      |\n",
            "|    time_elapsed       | 21        |\n",
            "|    total_timesteps    | 6000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1199      |\n",
            "|    policy_loss        | 1.62e+08  |\n",
            "|    reward             | 1327998.5 |\n",
            "|    std                | 0.99      |\n",
            "|    value_loss         | 1.8e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 1300      |\n",
            "|    time_elapsed       | 23        |\n",
            "|    total_timesteps    | 6500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.5     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1299      |\n",
            "|    policy_loss        | 1.94e+08  |\n",
            "|    reward             | 1601989.1 |\n",
            "|    std                | 0.99      |\n",
            "|    value_loss         | 2.76e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 1400      |\n",
            "|    time_elapsed       | 25        |\n",
            "|    total_timesteps    | 7000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | -2.38e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1399      |\n",
            "|    policy_loss        | 2.88e+08  |\n",
            "|    reward             | 2450167.8 |\n",
            "|    std                | 0.99      |\n",
            "|    value_loss         | 6.23e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 1500      |\n",
            "|    time_elapsed       | 26        |\n",
            "|    total_timesteps    | 7500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1499      |\n",
            "|    policy_loss        | 3.21e+08  |\n",
            "|    reward             | 2877680.2 |\n",
            "|    std                | 0.989     |\n",
            "|    value_loss         | 8.28e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 1600      |\n",
            "|    time_elapsed       | 28        |\n",
            "|    total_timesteps    | 8000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1599      |\n",
            "|    policy_loss        | 4.49e+08  |\n",
            "|    reward             | 3838685.2 |\n",
            "|    std                | 0.99      |\n",
            "|    value_loss         | 1.53e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 1700      |\n",
            "|    time_elapsed       | 30        |\n",
            "|    total_timesteps    | 8500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1699      |\n",
            "|    policy_loss        | 5.53e+08  |\n",
            "|    reward             | 4729506.0 |\n",
            "|    std                | 0.988     |\n",
            "|    value_loss         | 2.26e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4915936.396603354\n",
            "Sharpe:  0.8516287074638962\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 1800      |\n",
            "|    time_elapsed       | 32        |\n",
            "|    total_timesteps    | 9000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1799      |\n",
            "|    policy_loss        | 1.64e+08  |\n",
            "|    reward             | 1399163.4 |\n",
            "|    std                | 0.989     |\n",
            "|    value_loss         | 2e+13     |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 1900      |\n",
            "|    time_elapsed       | 34        |\n",
            "|    total_timesteps    | 9500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1899      |\n",
            "|    policy_loss        | 2.06e+08  |\n",
            "|    reward             | 1757966.8 |\n",
            "|    std                | 0.988     |\n",
            "|    value_loss         | 3.32e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 2000      |\n",
            "|    time_elapsed       | 35        |\n",
            "|    total_timesteps    | 10000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 1999      |\n",
            "|    policy_loss        | 2.94e+08  |\n",
            "|    reward             | 2339725.5 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 5.63e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 2100      |\n",
            "|    time_elapsed       | 37        |\n",
            "|    total_timesteps    | 10500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2099      |\n",
            "|    policy_loss        | 3.27e+08  |\n",
            "|    reward             | 2787623.2 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 8.1e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 2200      |\n",
            "|    time_elapsed       | 39        |\n",
            "|    total_timesteps    | 11000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2199      |\n",
            "|    policy_loss        | 4.45e+08  |\n",
            "|    reward             | 3666537.2 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 1.6e+14   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 2300      |\n",
            "|    time_elapsed       | 41        |\n",
            "|    total_timesteps    | 11500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2299      |\n",
            "|    policy_loss        | 4.35e+08  |\n",
            "|    reward             | 3467618.2 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 1.63e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4323323.4219220895\n",
            "Sharpe:  0.7921166508642308\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 2400      |\n",
            "|    time_elapsed       | 43        |\n",
            "|    total_timesteps    | 12000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2399      |\n",
            "|    policy_loss        | 1.67e+08  |\n",
            "|    reward             | 1407835.0 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 2.01e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 2500      |\n",
            "|    time_elapsed       | 45        |\n",
            "|    total_timesteps    | 12500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2499      |\n",
            "|    policy_loss        | 2.16e+08  |\n",
            "|    reward             | 1878467.2 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 3.65e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 2600      |\n",
            "|    time_elapsed       | 46        |\n",
            "|    total_timesteps    | 13000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2599      |\n",
            "|    policy_loss        | 3.38e+08  |\n",
            "|    reward             | 2697450.5 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 7.65e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 2700      |\n",
            "|    time_elapsed       | 48        |\n",
            "|    total_timesteps    | 13500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2699      |\n",
            "|    policy_loss        | 3.68e+08  |\n",
            "|    reward             | 3049030.0 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 1e+14     |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 278       |\n",
            "|    iterations         | 2800      |\n",
            "|    time_elapsed       | 50        |\n",
            "|    total_timesteps    | 14000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2799      |\n",
            "|    policy_loss        | 4.89e+08  |\n",
            "|    reward             | 4216674.5 |\n",
            "|    std                | 0.986     |\n",
            "|    value_loss         | 1.84e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4585091.979710431\n",
            "Sharpe:  0.8173410913363106\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 2900      |\n",
            "|    time_elapsed       | 52        |\n",
            "|    total_timesteps    | 14500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2899      |\n",
            "|    policy_loss        | 1.06e+08  |\n",
            "|    reward             | 851329.44 |\n",
            "|    std                | 0.986     |\n",
            "|    value_loss         | 8.61e+12  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 3000      |\n",
            "|    time_elapsed       | 54        |\n",
            "|    total_timesteps    | 15000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 2999      |\n",
            "|    policy_loss        | 2.12e+08  |\n",
            "|    reward             | 1644782.6 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 2.86e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 3100      |\n",
            "|    time_elapsed       | 55        |\n",
            "|    total_timesteps    | 15500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3099      |\n",
            "|    policy_loss        | 2.52e+08  |\n",
            "|    reward             | 2064740.2 |\n",
            "|    std                | 0.987     |\n",
            "|    value_loss         | 4.48e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 3200      |\n",
            "|    time_elapsed       | 57        |\n",
            "|    total_timesteps    | 16000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3199      |\n",
            "|    policy_loss        | 3.14e+08  |\n",
            "|    reward             | 2825500.0 |\n",
            "|    std                | 0.986     |\n",
            "|    value_loss         | 8.19e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 3300      |\n",
            "|    time_elapsed       | 59        |\n",
            "|    total_timesteps    | 16500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3299      |\n",
            "|    policy_loss        | 3.71e+08  |\n",
            "|    reward             | 3338794.2 |\n",
            "|    std                | 0.986     |\n",
            "|    value_loss         | 1.18e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 277       |\n",
            "|    iterations         | 3400      |\n",
            "|    time_elapsed       | 61        |\n",
            "|    total_timesteps    | 17000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3399      |\n",
            "|    policy_loss        | 4.71e+08  |\n",
            "|    reward             | 4277947.5 |\n",
            "|    std                | 0.985     |\n",
            "|    value_loss         | 1.92e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4717935.166093682\n",
            "Sharpe:  0.831934287369694\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 3500      |\n",
            "|    time_elapsed       | 63        |\n",
            "|    total_timesteps    | 17500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3499      |\n",
            "|    policy_loss        | 1.34e+08  |\n",
            "|    reward             | 1165116.6 |\n",
            "|    std                | 0.985     |\n",
            "|    value_loss         | 1.34e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 3600      |\n",
            "|    time_elapsed       | 65        |\n",
            "|    total_timesteps    | 18000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3599      |\n",
            "|    policy_loss        | 1.97e+08  |\n",
            "|    reward             | 1726830.0 |\n",
            "|    std                | 0.985     |\n",
            "|    value_loss         | 3.11e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 3700      |\n",
            "|    time_elapsed       | 66        |\n",
            "|    total_timesteps    | 18500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3699      |\n",
            "|    policy_loss        | 2.96e+08  |\n",
            "|    reward             | 2429737.0 |\n",
            "|    std                | 0.985     |\n",
            "|    value_loss         | 6.04e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 3800      |\n",
            "|    time_elapsed       | 68        |\n",
            "|    total_timesteps    | 19000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3799      |\n",
            "|    policy_loss        | 3.77e+08  |\n",
            "|    reward             | 3120817.5 |\n",
            "|    std                | 0.984     |\n",
            "|    value_loss         | 9.97e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 3900      |\n",
            "|    time_elapsed       | 70        |\n",
            "|    total_timesteps    | 19500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3899      |\n",
            "|    policy_loss        | 4.01e+08  |\n",
            "|    reward             | 3682035.8 |\n",
            "|    std                | 0.984     |\n",
            "|    value_loss         | 1.43e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 276       |\n",
            "|    iterations         | 4000      |\n",
            "|    time_elapsed       | 72        |\n",
            "|    total_timesteps    | 20000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 3999      |\n",
            "|    policy_loss        | 5.64e+08  |\n",
            "|    reward             | 4878761.0 |\n",
            "|    std                | 0.983     |\n",
            "|    value_loss         | 2.46e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4720664.690723379\n",
            "Sharpe:  0.8320054516158181\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4100      |\n",
            "|    time_elapsed       | 74        |\n",
            "|    total_timesteps    | 20500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4099      |\n",
            "|    policy_loss        | 1.6e+08   |\n",
            "|    reward             | 1410621.9 |\n",
            "|    std                | 0.983     |\n",
            "|    value_loss         | 2.04e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4200      |\n",
            "|    time_elapsed       | 76        |\n",
            "|    total_timesteps    | 21000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4199      |\n",
            "|    policy_loss        | 2.01e+08  |\n",
            "|    reward             | 1691376.4 |\n",
            "|    std                | 0.983     |\n",
            "|    value_loss         | 2.96e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4300      |\n",
            "|    time_elapsed       | 78        |\n",
            "|    total_timesteps    | 21500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4299      |\n",
            "|    policy_loss        | 2.74e+08  |\n",
            "|    reward             | 2511432.2 |\n",
            "|    std                | 0.982     |\n",
            "|    value_loss         | 6.69e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4400      |\n",
            "|    time_elapsed       | 79        |\n",
            "|    total_timesteps    | 22000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4399      |\n",
            "|    policy_loss        | 3.48e+08  |\n",
            "|    reward             | 3009766.5 |\n",
            "|    std                | 0.982     |\n",
            "|    value_loss         | 1.01e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4500      |\n",
            "|    time_elapsed       | 81        |\n",
            "|    total_timesteps    | 22500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4499      |\n",
            "|    policy_loss        | 4.63e+08  |\n",
            "|    reward             | 4148861.0 |\n",
            "|    std                | 0.981     |\n",
            "|    value_loss         | 1.81e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 275       |\n",
            "|    iterations         | 4600      |\n",
            "|    time_elapsed       | 83        |\n",
            "|    total_timesteps    | 23000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4599      |\n",
            "|    policy_loss        | 5.57e+08  |\n",
            "|    reward             | 5179502.0 |\n",
            "|    std                | 0.981     |\n",
            "|    value_loss         | 2.9e+14   |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4994902.1378531745\n",
            "Sharpe:  0.8562515997490507\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 274       |\n",
            "|    iterations         | 4700      |\n",
            "|    time_elapsed       | 85        |\n",
            "|    total_timesteps    | 23500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4699      |\n",
            "|    policy_loss        | 1.56e+08  |\n",
            "|    reward             | 1399054.8 |\n",
            "|    std                | 0.981     |\n",
            "|    value_loss         | 1.95e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 274       |\n",
            "|    iterations         | 4800      |\n",
            "|    time_elapsed       | 87        |\n",
            "|    total_timesteps    | 24000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4799      |\n",
            "|    policy_loss        | 2.1e+08   |\n",
            "|    reward             | 1824725.2 |\n",
            "|    std                | 0.981     |\n",
            "|    value_loss         | 3.55e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 274       |\n",
            "|    iterations         | 4900      |\n",
            "|    time_elapsed       | 89        |\n",
            "|    total_timesteps    | 24500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.2     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4899      |\n",
            "|    policy_loss        | 3.17e+08  |\n",
            "|    reward             | 2672124.8 |\n",
            "|    std                | 0.98      |\n",
            "|    value_loss         | 7.45e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 274       |\n",
            "|    iterations         | 5000      |\n",
            "|    time_elapsed       | 91        |\n",
            "|    total_timesteps    | 25000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 4999      |\n",
            "|    policy_loss        | 3.46e+08  |\n",
            "|    reward             | 3039430.2 |\n",
            "|    std                | 0.979     |\n",
            "|    value_loss         | 1.01e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 274       |\n",
            "|    iterations         | 5100      |\n",
            "|    time_elapsed       | 93        |\n",
            "|    total_timesteps    | 25500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5099      |\n",
            "|    policy_loss        | 4.48e+08  |\n",
            "|    reward             | 4305185.5 |\n",
            "|    std                | 0.978     |\n",
            "|    value_loss         | 1.88e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 273       |\n",
            "|    iterations         | 5200      |\n",
            "|    time_elapsed       | 94        |\n",
            "|    total_timesteps    | 26000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5199      |\n",
            "|    policy_loss        | 5.08e+08  |\n",
            "|    reward             | 4664920.0 |\n",
            "|    std                | 0.977     |\n",
            "|    value_loss         | 2.29e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4958351.328165317\n",
            "Sharpe:  0.8531998743258091\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 272       |\n",
            "|    iterations         | 5300      |\n",
            "|    time_elapsed       | 97        |\n",
            "|    total_timesteps    | 26500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5299      |\n",
            "|    policy_loss        | 1.73e+08  |\n",
            "|    reward             | 1545853.0 |\n",
            "|    std                | 0.977     |\n",
            "|    value_loss         | 2.48e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 272       |\n",
            "|    iterations         | 5400      |\n",
            "|    time_elapsed       | 99        |\n",
            "|    total_timesteps    | 27000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5399      |\n",
            "|    policy_loss        | 2.34e+08  |\n",
            "|    reward             | 1937584.8 |\n",
            "|    std                | 0.977     |\n",
            "|    value_loss         | 4.23e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 272       |\n",
            "|    iterations         | 5500      |\n",
            "|    time_elapsed       | 101       |\n",
            "|    total_timesteps    | 27500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39.1     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5499      |\n",
            "|    policy_loss        | 3.03e+08  |\n",
            "|    reward             | 2760013.0 |\n",
            "|    std                | 0.977     |\n",
            "|    value_loss         | 7.42e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 271       |\n",
            "|    iterations         | 5600      |\n",
            "|    time_elapsed       | 102       |\n",
            "|    total_timesteps    | 28000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5599      |\n",
            "|    policy_loss        | 3.52e+08  |\n",
            "|    reward             | 3033330.8 |\n",
            "|    std                | 0.976     |\n",
            "|    value_loss         | 9.92e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 271       |\n",
            "|    iterations         | 5700      |\n",
            "|    time_elapsed       | 104       |\n",
            "|    total_timesteps    | 28500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5699      |\n",
            "|    policy_loss        | 4.95e+08  |\n",
            "|    reward             | 4345211.0 |\n",
            "|    std                | 0.975     |\n",
            "|    value_loss         | 2.22e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:5026743.85358103\n",
            "Sharpe:  0.8625119086649907\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 270       |\n",
            "|    iterations         | 5800      |\n",
            "|    time_elapsed       | 107       |\n",
            "|    total_timesteps    | 29000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5799      |\n",
            "|    policy_loss        | 1.1e+08   |\n",
            "|    reward             | 999077.56 |\n",
            "|    std                | 0.975     |\n",
            "|    value_loss         | 9.88e+12  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 270       |\n",
            "|    iterations         | 5900      |\n",
            "|    time_elapsed       | 109       |\n",
            "|    total_timesteps    | 29500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5899      |\n",
            "|    policy_loss        | 1.89e+08  |\n",
            "|    reward             | 1661125.2 |\n",
            "|    std                | 0.975     |\n",
            "|    value_loss         | 2.91e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 269       |\n",
            "|    iterations         | 6000      |\n",
            "|    time_elapsed       | 111       |\n",
            "|    total_timesteps    | 30000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 5999      |\n",
            "|    policy_loss        | 2.44e+08  |\n",
            "|    reward             | 2124258.8 |\n",
            "|    std                | 0.974     |\n",
            "|    value_loss         | 4.85e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 269       |\n",
            "|    iterations         | 6100      |\n",
            "|    time_elapsed       | 113       |\n",
            "|    total_timesteps    | 30500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6099      |\n",
            "|    policy_loss        | 3.34e+08  |\n",
            "|    reward             | 2953049.0 |\n",
            "|    std                | 0.974     |\n",
            "|    value_loss         | 9.67e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 269       |\n",
            "|    iterations         | 6200      |\n",
            "|    time_elapsed       | 115       |\n",
            "|    total_timesteps    | 31000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6199      |\n",
            "|    policy_loss        | 4e+08     |\n",
            "|    reward             | 3546849.0 |\n",
            "|    std                | 0.974     |\n",
            "|    value_loss         | 1.38e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 269       |\n",
            "|    iterations         | 6300      |\n",
            "|    time_elapsed       | 117       |\n",
            "|    total_timesteps    | 31500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -39       |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6299      |\n",
            "|    policy_loss        | 5.51e+08  |\n",
            "|    reward             | 4712991.0 |\n",
            "|    std                | 0.973     |\n",
            "|    value_loss         | 2.28e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4837376.482801431\n",
            "Sharpe:  0.8387173422743931\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 268       |\n",
            "|    iterations         | 6400      |\n",
            "|    time_elapsed       | 119       |\n",
            "|    total_timesteps    | 32000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6399      |\n",
            "|    policy_loss        | 1.38e+08  |\n",
            "|    reward             | 1211580.4 |\n",
            "|    std                | 0.973     |\n",
            "|    value_loss         | 1.45e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 268       |\n",
            "|    iterations         | 6500      |\n",
            "|    time_elapsed       | 121       |\n",
            "|    total_timesteps    | 32500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6499      |\n",
            "|    policy_loss        | 1.91e+08  |\n",
            "|    reward             | 1477593.8 |\n",
            "|    std                | 0.972     |\n",
            "|    value_loss         | 2.54e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 267       |\n",
            "|    iterations         | 6600      |\n",
            "|    time_elapsed       | 123       |\n",
            "|    total_timesteps    | 33000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6599      |\n",
            "|    policy_loss        | 2.73e+08  |\n",
            "|    reward             | 2254119.2 |\n",
            "|    std                | 0.971     |\n",
            "|    value_loss         | 5.41e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 267       |\n",
            "|    iterations         | 6700      |\n",
            "|    time_elapsed       | 125       |\n",
            "|    total_timesteps    | 33500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6699      |\n",
            "|    policy_loss        | 3.51e+08  |\n",
            "|    reward             | 2718629.5 |\n",
            "|    std                | 0.971     |\n",
            "|    value_loss         | 8.09e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 267       |\n",
            "|    iterations         | 6800      |\n",
            "|    time_elapsed       | 127       |\n",
            "|    total_timesteps    | 34000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6799      |\n",
            "|    policy_loss        | 4.09e+08  |\n",
            "|    reward             | 3603140.0 |\n",
            "|    std                | 0.97      |\n",
            "|    value_loss         | 1.38e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 267       |\n",
            "|    iterations         | 6900      |\n",
            "|    time_elapsed       | 129       |\n",
            "|    total_timesteps    | 34500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6899      |\n",
            "|    policy_loss        | 5.3e+08   |\n",
            "|    reward             | 4691950.0 |\n",
            "|    std                | 0.97      |\n",
            "|    value_loss         | 2.27e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4853781.912040991\n",
            "Sharpe:  0.844467759049473\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 266       |\n",
            "|    iterations         | 7000      |\n",
            "|    time_elapsed       | 131       |\n",
            "|    total_timesteps    | 35000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 6999      |\n",
            "|    policy_loss        | 1.46e+08  |\n",
            "|    reward             | 1312820.6 |\n",
            "|    std                | 0.97      |\n",
            "|    value_loss         | 1.72e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 266       |\n",
            "|    iterations         | 7100      |\n",
            "|    time_elapsed       | 133       |\n",
            "|    total_timesteps    | 35500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.9     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7099      |\n",
            "|    policy_loss        | 2.02e+08  |\n",
            "|    reward             | 1782995.5 |\n",
            "|    std                | 0.969     |\n",
            "|    value_loss         | 3.29e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 266       |\n",
            "|    iterations         | 7200      |\n",
            "|    time_elapsed       | 135       |\n",
            "|    total_timesteps    | 36000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7199      |\n",
            "|    policy_loss        | 2.75e+08  |\n",
            "|    reward             | 2355707.5 |\n",
            "|    std                | 0.969     |\n",
            "|    value_loss         | 5.75e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 265       |\n",
            "|    iterations         | 7300      |\n",
            "|    time_elapsed       | 137       |\n",
            "|    total_timesteps    | 36500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7299      |\n",
            "|    policy_loss        | 3.01e+08  |\n",
            "|    reward             | 2606537.8 |\n",
            "|    std                | 0.969     |\n",
            "|    value_loss         | 7.08e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 265       |\n",
            "|    iterations         | 7400      |\n",
            "|    time_elapsed       | 139       |\n",
            "|    total_timesteps    | 37000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7399      |\n",
            "|    policy_loss        | 4.64e+08  |\n",
            "|    reward             | 4004034.5 |\n",
            "|    std                | 0.967     |\n",
            "|    value_loss         | 1.71e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 265       |\n",
            "|    iterations         | 7500      |\n",
            "|    time_elapsed       | 141       |\n",
            "|    total_timesteps    | 37500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | -2.38e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7499      |\n",
            "|    policy_loss        | 5.32e+08  |\n",
            "|    reward             | 4737323.0 |\n",
            "|    std                | 0.967     |\n",
            "|    value_loss         | 2.5e+14   |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4493241.386379412\n",
            "Sharpe:  0.8079884443208769\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 264       |\n",
            "|    iterations         | 7600      |\n",
            "|    time_elapsed       | 143       |\n",
            "|    total_timesteps    | 38000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7599      |\n",
            "|    policy_loss        | 1.43e+08  |\n",
            "|    reward             | 1339165.2 |\n",
            "|    std                | 0.966     |\n",
            "|    value_loss         | 1.84e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 264       |\n",
            "|    iterations         | 7700      |\n",
            "|    time_elapsed       | 145       |\n",
            "|    total_timesteps    | 38500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7699      |\n",
            "|    policy_loss        | 2.02e+08  |\n",
            "|    reward             | 1782240.8 |\n",
            "|    std                | 0.966     |\n",
            "|    value_loss         | 3.35e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 263       |\n",
            "|    iterations         | 7800      |\n",
            "|    time_elapsed       | 147       |\n",
            "|    total_timesteps    | 39000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.8     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7799      |\n",
            "|    policy_loss        | 2.97e+08  |\n",
            "|    reward             | 2583803.8 |\n",
            "|    std                | 0.966     |\n",
            "|    value_loss         | 7.11e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 263       |\n",
            "|    iterations         | 7900      |\n",
            "|    time_elapsed       | 149       |\n",
            "|    total_timesteps    | 39500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7899      |\n",
            "|    policy_loss        | 3.43e+08  |\n",
            "|    reward             | 2971294.2 |\n",
            "|    std                | 0.966     |\n",
            "|    value_loss         | 8.78e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 263       |\n",
            "|    iterations         | 8000      |\n",
            "|    time_elapsed       | 151       |\n",
            "|    total_timesteps    | 40000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 7999      |\n",
            "|    policy_loss        | 4.98e+08  |\n",
            "|    reward             | 3982222.0 |\n",
            "|    std                | 0.965     |\n",
            "|    value_loss         | 1.7e+14   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 262       |\n",
            "|    iterations         | 8100      |\n",
            "|    time_elapsed       | 154       |\n",
            "|    total_timesteps    | 40500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8099      |\n",
            "|    policy_loss        | 5.7e+08   |\n",
            "|    reward             | 4661376.5 |\n",
            "|    std                | 0.965     |\n",
            "|    value_loss         | 2.42e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4713909.819186837\n",
            "Sharpe:  0.8374218582541628\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 262       |\n",
            "|    iterations         | 8200      |\n",
            "|    time_elapsed       | 156       |\n",
            "|    total_timesteps    | 41000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8199      |\n",
            "|    policy_loss        | 1.81e+08  |\n",
            "|    reward             | 1572754.4 |\n",
            "|    std                | 0.965     |\n",
            "|    value_loss         | 2.61e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 262       |\n",
            "|    iterations         | 8300      |\n",
            "|    time_elapsed       | 158       |\n",
            "|    total_timesteps    | 41500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8299      |\n",
            "|    policy_loss        | 2.3e+08   |\n",
            "|    reward             | 1943967.4 |\n",
            "|    std                | 0.964     |\n",
            "|    value_loss         | 3.98e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 261       |\n",
            "|    iterations         | 8400      |\n",
            "|    time_elapsed       | 160       |\n",
            "|    total_timesteps    | 42000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8399      |\n",
            "|    policy_loss        | 3.15e+08  |\n",
            "|    reward             | 2741592.8 |\n",
            "|    std                | 0.964     |\n",
            "|    value_loss         | 8.72e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 261       |\n",
            "|    iterations         | 8500      |\n",
            "|    time_elapsed       | 162       |\n",
            "|    total_timesteps    | 42500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8499      |\n",
            "|    policy_loss        | 3.4e+08   |\n",
            "|    reward             | 3239286.5 |\n",
            "|    std                | 0.963     |\n",
            "|    value_loss         | 1.08e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 261       |\n",
            "|    iterations         | 8600      |\n",
            "|    time_elapsed       | 164       |\n",
            "|    total_timesteps    | 43000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.7     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8599      |\n",
            "|    policy_loss        | 4.83e+08  |\n",
            "|    reward             | 4284561.5 |\n",
            "|    std                | 0.963     |\n",
            "|    value_loss         | 1.88e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4523811.030662169\n",
            "Sharpe:  0.8126144595072541\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 260       |\n",
            "|    iterations         | 8700      |\n",
            "|    time_elapsed       | 166       |\n",
            "|    total_timesteps    | 43500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8699      |\n",
            "|    policy_loss        | 1.2e+08   |\n",
            "|    reward             | 1067354.1 |\n",
            "|    std                | 0.962     |\n",
            "|    value_loss         | 1.1e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 260       |\n",
            "|    iterations         | 8800      |\n",
            "|    time_elapsed       | 168       |\n",
            "|    total_timesteps    | 44000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8799      |\n",
            "|    policy_loss        | 1.8e+08   |\n",
            "|    reward             | 1622284.8 |\n",
            "|    std                | 0.961     |\n",
            "|    value_loss         | 2.85e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 260       |\n",
            "|    iterations         | 8900      |\n",
            "|    time_elapsed       | 171       |\n",
            "|    total_timesteps    | 44500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8899      |\n",
            "|    policy_loss        | 2.59e+08  |\n",
            "|    reward             | 2209715.5 |\n",
            "|    std                | 0.961     |\n",
            "|    value_loss         | 5.19e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 259       |\n",
            "|    iterations         | 9000      |\n",
            "|    time_elapsed       | 173       |\n",
            "|    total_timesteps    | 45000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 8999      |\n",
            "|    policy_loss        | 3.12e+08  |\n",
            "|    reward             | 2949269.0 |\n",
            "|    std                | 0.96      |\n",
            "|    value_loss         | 9.1e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 259       |\n",
            "|    iterations         | 9100      |\n",
            "|    time_elapsed       | 175       |\n",
            "|    total_timesteps    | 45500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9099      |\n",
            "|    policy_loss        | 4.37e+08  |\n",
            "|    reward             | 3359352.2 |\n",
            "|    std                | 0.961     |\n",
            "|    value_loss         | 1.21e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 259       |\n",
            "|    iterations         | 9200      |\n",
            "|    time_elapsed       | 177       |\n",
            "|    total_timesteps    | 46000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9199      |\n",
            "|    policy_loss        | 4.52e+08  |\n",
            "|    reward             | 4211900.5 |\n",
            "|    std                | 0.96      |\n",
            "|    value_loss         | 1.96e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4291701.430059653\n",
            "Sharpe:  0.7902905712690268\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 258       |\n",
            "|    iterations         | 9300      |\n",
            "|    time_elapsed       | 179       |\n",
            "|    total_timesteps    | 46500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9299      |\n",
            "|    policy_loss        | 1.39e+08  |\n",
            "|    reward             | 1250208.5 |\n",
            "|    std                | 0.96      |\n",
            "|    value_loss         | 1.7e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 258       |\n",
            "|    iterations         | 9400      |\n",
            "|    time_elapsed       | 182       |\n",
            "|    total_timesteps    | 47000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 2.38e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9399      |\n",
            "|    policy_loss        | 1.76e+08  |\n",
            "|    reward             | 1676908.4 |\n",
            "|    std                | 0.96      |\n",
            "|    value_loss         | 2.7e+13   |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 257       |\n",
            "|    iterations         | 9500      |\n",
            "|    time_elapsed       | 184       |\n",
            "|    total_timesteps    | 47500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9499      |\n",
            "|    policy_loss        | 2.55e+08  |\n",
            "|    reward             | 2381557.8 |\n",
            "|    std                | 0.959     |\n",
            "|    value_loss         | 5.85e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 257       |\n",
            "|    iterations         | 9600      |\n",
            "|    time_elapsed       | 186       |\n",
            "|    total_timesteps    | 48000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.6     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9599      |\n",
            "|    policy_loss        | 3.08e+08  |\n",
            "|    reward             | 2836898.5 |\n",
            "|    std                | 0.959     |\n",
            "|    value_loss         | 8.46e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 257       |\n",
            "|    iterations         | 9700      |\n",
            "|    time_elapsed       | 188       |\n",
            "|    total_timesteps    | 48500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.5     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9699      |\n",
            "|    policy_loss        | 4.47e+08  |\n",
            "|    reward             | 3745362.0 |\n",
            "|    std                | 0.959     |\n",
            "|    value_loss         | 1.49e+14  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 257       |\n",
            "|    iterations         | 9800      |\n",
            "|    time_elapsed       | 190       |\n",
            "|    total_timesteps    | 49000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.5     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9799      |\n",
            "|    policy_loss        | 5.56e+08  |\n",
            "|    reward             | 4747769.5 |\n",
            "|    std                | 0.958     |\n",
            "|    value_loss         | 2.33e+14  |\n",
            "-------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4621059.893247926\n",
            "Sharpe:  0.829967238337267\n",
            "=================================\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 256       |\n",
            "|    iterations         | 9900      |\n",
            "|    time_elapsed       | 192       |\n",
            "|    total_timesteps    | 49500     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.5     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9899      |\n",
            "|    policy_loss        | 1.6e+08   |\n",
            "|    reward             | 1455380.1 |\n",
            "|    std                | 0.958     |\n",
            "|    value_loss         | 2.22e+13  |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 256       |\n",
            "|    iterations         | 10000     |\n",
            "|    time_elapsed       | 195       |\n",
            "|    total_timesteps    | 50000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -38.5     |\n",
            "|    explained_variance | 1.79e-07  |\n",
            "|    learning_rate      | 0.0002    |\n",
            "|    n_updates          | 9999      |\n",
            "|    policy_loss        | 2.09e+08  |\n",
            "|    reward             | 1902459.8 |\n",
            "|    std                | 0.958     |\n",
            "|    value_loss         | 3.79e+13  |\n",
            "-------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cfhURI26P_Nr",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_a2c.save('/content/trained_models/trained_a2c.zip')"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvrqTro3lhAh"
      },
      "source": [
        "### Model 2: **PPO**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kVXta7jVKMhV",
        "outputId": "619c0909-eb34-43d5-bcd6-3a78ec4c74aa",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "PPO_PARAMS = {\n",
        "    \"n_steps\": 2048,\n",
        "    \"ent_coef\": 0.005,\n",
        "    \"learning_rate\": 0.0001,\n",
        "    \"batch_size\": 128,\n",
        "}\n",
        "model_ppo = agent.get_model(\"ppo\",model_kwargs = PPO_PARAMS)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'n_steps': 2048, 'ent_coef': 0.005, 'learning_rate': 0.0001, 'batch_size': 128}\n",
            "Using cpu device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z5XlUIszKUGx",
        "outputId": "974eec9d-10a9-422d-cc35-0318743bf91a",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_ppo = agent.train_model(model=model_ppo, \n",
        "                             tb_log_name='ppo',\n",
        "                             total_timesteps=80000)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    fps             | 399       |\n",
            "|    iterations      | 1         |\n",
            "|    time_elapsed    | 5         |\n",
            "|    total_timesteps | 2048      |\n",
            "| train/             |           |\n",
            "|    reward          | 3486273.2 |\n",
            "----------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4649806.666052939\n",
            "Sharpe:  0.8211911179029259\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 354       |\n",
            "|    iterations           | 2         |\n",
            "|    time_elapsed         | 11        |\n",
            "|    total_timesteps      | 4096      |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 7.37e+14  |\n",
            "|    n_updates            | 10        |\n",
            "|    policy_gradient_loss | -7.01e-07 |\n",
            "|    reward               | 2397798.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.4e+15   |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4680718.31445438\n",
            "Sharpe:  0.8272293369087982\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 344       |\n",
            "|    iterations           | 3         |\n",
            "|    time_elapsed         | 17        |\n",
            "|    total_timesteps      | 6144      |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 5.96e-08  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.22e+15  |\n",
            "|    n_updates            | 20        |\n",
            "|    policy_gradient_loss | -5.43e-07 |\n",
            "|    reward               | 1277743.6 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.5e+15   |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 343       |\n",
            "|    iterations           | 4         |\n",
            "|    time_elapsed         | 23        |\n",
            "|    total_timesteps      | 8192      |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.58e+15  |\n",
            "|    n_updates            | 30        |\n",
            "|    policy_gradient_loss | -3.48e-07 |\n",
            "|    reward               | 4121737.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 3.23e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4497386.573334347\n",
            "Sharpe:  0.8094982851779371\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 339       |\n",
            "|    iterations           | 5         |\n",
            "|    time_elapsed         | 30        |\n",
            "|    total_timesteps      | 10240     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.8e+14   |\n",
            "|    n_updates            | 40        |\n",
            "|    policy_gradient_loss | -7.13e-07 |\n",
            "|    reward               | 2706885.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.91e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4517981.860568596\n",
            "Sharpe:  0.8095856346629304\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 336       |\n",
            "|    iterations           | 6         |\n",
            "|    time_elapsed         | 36        |\n",
            "|    total_timesteps      | 12288     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.1e+15   |\n",
            "|    n_updates            | 50        |\n",
            "|    policy_gradient_loss | -5.19e-07 |\n",
            "|    reward               | 1617559.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.91e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 336       |\n",
            "|    iterations           | 7         |\n",
            "|    time_elapsed         | 42        |\n",
            "|    total_timesteps      | 14336     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.25e+15  |\n",
            "|    n_updates            | 60        |\n",
            "|    policy_gradient_loss | -4.81e-07 |\n",
            "|    reward               | 4972709.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.59e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4719288.005276221\n",
            "Sharpe:  0.8286201866479401\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 333       |\n",
            "|    iterations           | 8         |\n",
            "|    time_elapsed         | 49        |\n",
            "|    total_timesteps      | 16384     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.79e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.42e+15  |\n",
            "|    n_updates            | 70        |\n",
            "|    policy_gradient_loss | -5.53e-07 |\n",
            "|    reward               | 3013871.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.78e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4243715.197853623\n",
            "Sharpe:  0.7808670428066317\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 332       |\n",
            "|    iterations           | 9         |\n",
            "|    time_elapsed         | 55        |\n",
            "|    total_timesteps      | 18432     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -2.38e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 7.34e+14  |\n",
            "|    n_updates            | 80        |\n",
            "|    policy_gradient_loss | -7.41e-07 |\n",
            "|    reward               | 2360915.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.41e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:5062374.027912245\n",
            "Sharpe:  0.8621370571756811\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 331       |\n",
            "|    iterations           | 10        |\n",
            "|    time_elapsed         | 61        |\n",
            "|    total_timesteps      | 20480     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 5.96e-08  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.16e+15  |\n",
            "|    n_updates            | 90        |\n",
            "|    policy_gradient_loss | -5.92e-07 |\n",
            "|    reward               | 1343733.1 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.44e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 331       |\n",
            "|    iterations           | 11        |\n",
            "|    time_elapsed         | 67        |\n",
            "|    total_timesteps      | 22528     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.86e+15  |\n",
            "|    n_updates            | 100       |\n",
            "|    policy_gradient_loss | -3.83e-07 |\n",
            "|    reward               | 4218010.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 3.62e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4647120.390608123\n",
            "Sharpe:  0.820766118907127\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 331       |\n",
            "|    iterations           | 12        |\n",
            "|    time_elapsed         | 74        |\n",
            "|    total_timesteps      | 24576     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.42e+14  |\n",
            "|    n_updates            | 110       |\n",
            "|    policy_gradient_loss | -5.61e-07 |\n",
            "|    reward               | 2692974.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.77e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4533650.921120629\n",
            "Sharpe:  0.8103216178688569\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 330       |\n",
            "|    iterations           | 13        |\n",
            "|    time_elapsed         | 80        |\n",
            "|    total_timesteps      | 26624     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.07e+15  |\n",
            "|    n_updates            | 120       |\n",
            "|    policy_gradient_loss | -4.7e-07  |\n",
            "|    reward               | 1683982.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.19e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 331       |\n",
            "|    iterations           | 14        |\n",
            "|    time_elapsed         | 86        |\n",
            "|    total_timesteps      | 28672     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.47e+15  |\n",
            "|    n_updates            | 130       |\n",
            "|    policy_gradient_loss | -3.27e-07 |\n",
            "|    reward               | 4282457.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 3e+15     |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4297634.600608397\n",
            "Sharpe:  0.7855784287966721\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 330       |\n",
            "|    iterations           | 15        |\n",
            "|    time_elapsed         | 93        |\n",
            "|    total_timesteps      | 30720     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.19e+15  |\n",
            "|    n_updates            | 140       |\n",
            "|    policy_gradient_loss | -4.51e-07 |\n",
            "|    reward               | 2598395.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.28e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4807937.647553395\n",
            "Sharpe:  0.8416587685342838\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 329       |\n",
            "|    iterations           | 16        |\n",
            "|    time_elapsed         | 99        |\n",
            "|    total_timesteps      | 32768     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 5.96e-08  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 8.08e+14  |\n",
            "|    n_updates            | 150       |\n",
            "|    policy_gradient_loss | -6.93e-07 |\n",
            "|    reward               | 1840154.4 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.6e+15   |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4298507.176847542\n",
            "Sharpe:  0.788247840121946\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 328       |\n",
            "|    iterations           | 17        |\n",
            "|    time_elapsed         | 106       |\n",
            "|    total_timesteps      | 34816     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.3e+15   |\n",
            "|    n_updates            | 160       |\n",
            "|    policy_gradient_loss | -5.15e-07 |\n",
            "|    reward               | 1057142.6 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.64e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 323       |\n",
            "|    iterations           | 18        |\n",
            "|    time_elapsed         | 113       |\n",
            "|    total_timesteps      | 36864     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.4e+15   |\n",
            "|    n_updates            | 170       |\n",
            "|    policy_gradient_loss | -5.2e-07  |\n",
            "|    reward               | 3761609.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.81e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:5112031.0673039155\n",
            "Sharpe:  0.8696261088609344\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 19        |\n",
            "|    time_elapsed         | 120       |\n",
            "|    total_timesteps      | 38912     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 8.38e+14  |\n",
            "|    n_updates            | 180       |\n",
            "|    policy_gradient_loss | -6.46e-07 |\n",
            "|    reward               | 2437071.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.65e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4388713.02216537\n",
            "Sharpe:  0.7978631224961166\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 20        |\n",
            "|    time_elapsed         | 127       |\n",
            "|    total_timesteps      | 40960     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.31e+15  |\n",
            "|    n_updates            | 190       |\n",
            "|    policy_gradient_loss | -5.48e-07 |\n",
            "|    reward               | 1483517.9 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.65e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 21        |\n",
            "|    time_elapsed         | 133       |\n",
            "|    total_timesteps      | 43008     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.38e+15  |\n",
            "|    n_updates            | 200       |\n",
            "|    policy_gradient_loss | -3.77e-07 |\n",
            "|    reward               | 4002365.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.8e+15   |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4516153.892330541\n",
            "Sharpe:  0.809897592618647\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 22        |\n",
            "|    time_elapsed         | 140       |\n",
            "|    total_timesteps      | 45056     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.17e+15  |\n",
            "|    n_updates            | 210       |\n",
            "|    policy_gradient_loss | -5.42e-07 |\n",
            "|    reward               | 2940158.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.23e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4933619.366733749\n",
            "Sharpe:  0.8564381825301831\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 23        |\n",
            "|    time_elapsed         | 146       |\n",
            "|    total_timesteps      | 47104     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.79e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.38e+14  |\n",
            "|    n_updates            | 220       |\n",
            "|    policy_gradient_loss | -5.75e-07 |\n",
            "|    reward               | 1840265.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.91e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 24        |\n",
            "|    time_elapsed         | 152       |\n",
            "|    total_timesteps      | 49152     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.51e+15  |\n",
            "|    n_updates            | 230       |\n",
            "|    policy_gradient_loss | -3.21e-07 |\n",
            "|    reward               | 4345465.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.92e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4503615.4745328855\n",
            "Sharpe:  0.8065034255133612\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 25        |\n",
            "|    time_elapsed         | 159       |\n",
            "|    total_timesteps      | 51200     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.51e+15  |\n",
            "|    n_updates            | 240       |\n",
            "|    policy_gradient_loss | -3.07e-07 |\n",
            "|    reward               | 3443149.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.96e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4825457.135459208\n",
            "Sharpe:  0.8400920196479832\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 26        |\n",
            "|    time_elapsed         | 165       |\n",
            "|    total_timesteps      | 53248     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 7.12e+14  |\n",
            "|    n_updates            | 250       |\n",
            "|    policy_gradient_loss | -7.33e-07 |\n",
            "|    reward               | 2081357.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.45e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4328557.43270105\n",
            "Sharpe:  0.791595823998115\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 27        |\n",
            "|    time_elapsed         | 172       |\n",
            "|    total_timesteps      | 55296     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.44e+15  |\n",
            "|    n_updates            | 260       |\n",
            "|    policy_gradient_loss | -3.49e-07 |\n",
            "|    reward               | 1453382.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.71e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 28        |\n",
            "|    time_elapsed         | 178       |\n",
            "|    total_timesteps      | 57344     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.28e+15  |\n",
            "|    n_updates            | 270       |\n",
            "|    policy_gradient_loss | -4.58e-07 |\n",
            "|    reward               | 4066431.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.74e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4654883.526037905\n",
            "Sharpe:  0.8229863385178353\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 29        |\n",
            "|    time_elapsed         | 184       |\n",
            "|    total_timesteps      | 59392     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.8e+14   |\n",
            "|    n_updates            | 280       |\n",
            "|    policy_gradient_loss | -4.42e-07 |\n",
            "|    reward               | 2657798.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.87e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4359331.030059856\n",
            "Sharpe:  0.7985428162084375\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 321       |\n",
            "|    iterations           | 30        |\n",
            "|    time_elapsed         | 190       |\n",
            "|    total_timesteps      | 61440     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.31e+14  |\n",
            "|    n_updates            | 290       |\n",
            "|    policy_gradient_loss | -4.35e-07 |\n",
            "|    reward               | 1480231.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.03e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 31        |\n",
            "|    time_elapsed         | 197       |\n",
            "|    total_timesteps      | 63488     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.18e+15  |\n",
            "|    n_updates            | 300       |\n",
            "|    policy_gradient_loss | -7.31e-07 |\n",
            "|    reward               | 4919555.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.46e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4564109.466492769\n",
            "Sharpe:  0.810996999582129\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 32        |\n",
            "|    time_elapsed         | 203       |\n",
            "|    total_timesteps      | 65536     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.79e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.46e+15  |\n",
            "|    n_updates            | 310       |\n",
            "|    policy_gradient_loss | -4.78e-07 |\n",
            "|    reward               | 2766582.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.82e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4207895.853489561\n",
            "Sharpe:  0.7772078219545542\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 33        |\n",
            "|    time_elapsed         | 209       |\n",
            "|    total_timesteps      | 67584     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 6.56e+14  |\n",
            "|    n_updates            | 320       |\n",
            "|    policy_gradient_loss | -6.92e-07 |\n",
            "|    reward               | 2068131.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.41e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4597592.856398027\n",
            "Sharpe:  0.8184496399706171\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 34        |\n",
            "|    time_elapsed         | 216       |\n",
            "|    total_timesteps      | 69632     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.01e+15  |\n",
            "|    n_updates            | 330       |\n",
            "|    policy_gradient_loss | -4.74e-07 |\n",
            "|    reward               | 1279538.2 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.17e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 35        |\n",
            "|    time_elapsed         | 222       |\n",
            "|    total_timesteps      | 71680     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.57e+15  |\n",
            "|    n_updates            | 340       |\n",
            "|    policy_gradient_loss | -4.1e-07  |\n",
            "|    reward               | 3842762.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 3.18e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4482190.6865420835\n",
            "Sharpe:  0.8059341739414112\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 36        |\n",
            "|    time_elapsed         | 228       |\n",
            "|    total_timesteps      | 73728     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 7.7e+14   |\n",
            "|    n_updates            | 350       |\n",
            "|    policy_gradient_loss | -5.57e-07 |\n",
            "|    reward               | 2553034.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.58e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:5061586.181033656\n",
            "Sharpe:  0.861678320788483\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 37        |\n",
            "|    time_elapsed         | 235       |\n",
            "|    total_timesteps      | 75776     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | -1.19e-07 |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.89e+14  |\n",
            "|    n_updates            | 360       |\n",
            "|    policy_gradient_loss | -6.63e-07 |\n",
            "|    reward               | 1591508.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.15e+15  |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 38        |\n",
            "|    time_elapsed         | 241       |\n",
            "|    total_timesteps      | 77824     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 1.19e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.55e+15  |\n",
            "|    n_updates            | 370       |\n",
            "|    policy_gradient_loss | -4.5e-07  |\n",
            "|    reward               | 4252701.5 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 3.08e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4540656.773320585\n",
            "Sharpe:  0.8144111550959243\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 39        |\n",
            "|    time_elapsed         | 247       |\n",
            "|    total_timesteps      | 79872     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 0         |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 1.2e+15   |\n",
            "|    n_updates            | 380       |\n",
            "|    policy_gradient_loss | -4.05e-07 |\n",
            "|    reward               | 2811643.8 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 2.37e+15  |\n",
            "---------------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4302446.033189798\n",
            "Sharpe:  0.7866478229864164\n",
            "=================================\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 322       |\n",
            "|    iterations           | 40        |\n",
            "|    time_elapsed         | 253       |\n",
            "|    total_timesteps      | 81920     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0       |\n",
            "|    clip_fraction        | 0         |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -39.7     |\n",
            "|    explained_variance   | 2.38e-07  |\n",
            "|    learning_rate        | 0.0001    |\n",
            "|    loss                 | 9.01e+14  |\n",
            "|    n_updates            | 390       |\n",
            "|    policy_gradient_loss | -6.79e-07 |\n",
            "|    reward               | 1903558.0 |\n",
            "|    std                  | 1         |\n",
            "|    value_loss           | 1.71e+15  |\n",
            "---------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zmMmk1amUlEm",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_ppo.save('/content/trained_models/trained_ppo.zip')"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a3Iuv554xYFH"
      },
      "source": [
        "### Model 3: **DDPG**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ojmppgo4LPLz",
        "outputId": "1378d445-f15c-4b2c-833e-1f84e0d78ae2",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "DDPG_PARAMS = {\"batch_size\": 128, \"buffer_size\": 50000, \"learning_rate\": 0.001}\n",
        "\n",
        "\n",
        "model_ddpg = agent.get_model(\"ddpg\",model_kwargs = DDPG_PARAMS)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'batch_size': 128, 'buffer_size': 50000, 'learning_rate': 0.001}\n",
            "Using cpu device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "bWt6BIR0LT25",
        "outputId": "1cd63fbf-b7d1-4f3f-e410-77c383fb6406",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_ddpg = agent.train_model(model=model_ddpg, \n",
        "                             tb_log_name='ddpg',\n",
        "                             total_timesteps=50000)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4333815.787556955\n",
            "Sharpe:  0.789707272389417\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    episodes        | 4         |\n",
            "|    fps             | 23        |\n",
            "|    time_elapsed    | 490       |\n",
            "|    total_timesteps | 11572     |\n",
            "| train/             |           |\n",
            "|    actor_loss      | -7.88e+07 |\n",
            "|    critic_loss     | 5.33e+12  |\n",
            "|    learning_rate   | 0.001     |\n",
            "|    n_updates       | 8679      |\n",
            "|    reward          | 4389548.5 |\n",
            "----------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    episodes        | 8         |\n",
            "|    fps             | 20        |\n",
            "|    time_elapsed    | 1145      |\n",
            "|    total_timesteps | 23144     |\n",
            "| train/             |           |\n",
            "|    actor_loss      | -1.62e+08 |\n",
            "|    critic_loss     | 1.56e+13  |\n",
            "|    learning_rate   | 0.001     |\n",
            "|    n_updates       | 20251     |\n",
            "|    reward          | 4389548.5 |\n",
            "----------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    episodes        | 12        |\n",
            "|    fps             | 18        |\n",
            "|    time_elapsed    | 1840      |\n",
            "|    total_timesteps | 34716     |\n",
            "| train/             |           |\n",
            "|    actor_loss      | -2.08e+08 |\n",
            "|    critic_loss     | 1.75e+13  |\n",
            "|    learning_rate   | 0.001     |\n",
            "|    n_updates       | 31823     |\n",
            "|    reward          | 4389548.5 |\n",
            "----------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4389548.351760708\n",
            "Sharpe:  0.7962409611560134\n",
            "=================================\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-28-87b09b7794a1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m trained_ddpg = agent.train_model(model=model_ddpg, \n\u001b[1;32m      2\u001b[0m                              \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'ddpg'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m                              total_timesteps=50000)\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/finrl/drl_agents/stablebaselines3/models.py\u001b[0m in \u001b[0;36mtrain_model\u001b[0;34m(self, model, tb_log_name, total_timesteps)\u001b[0m\n\u001b[1;32m    103\u001b[0m             \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 105\u001b[0;31m             \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTensorboardCallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    106\u001b[0m         )\n\u001b[1;32m    107\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/ddpg/ddpg.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    137\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    138\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 139\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    140\u001b[0m         )\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/td3/td3.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    211\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         )\n\u001b[1;32m    215\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/off_policy_algorithm.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    369\u001b[0m                 \u001b[0;31m# Special case when the user passes `gradient_steps=0`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    370\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mgradient_steps\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 371\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgradient_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    372\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    373\u001b[0m         \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_training_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/td3/td3.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, gradient_steps, batch_size)\u001b[0m\n\u001b[1;32m    179\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m                 \u001b[0mactor_loss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 181\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    182\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m                 \u001b[0mpolyak_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcritic\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcritic_target\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtau\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     86\u001b[0m                 \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                     \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py\u001b[0m in \u001b[0;36mdecorate_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     26\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    142\u001b[0m                    \u001b[0mlr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'lr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m                    \u001b[0mweight_decay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'weight_decay'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m                    eps=group['eps'])\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py\u001b[0m in \u001b[0;36madam\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps)\u001b[0m\n\u001b[1;32m     92\u001b[0m             \u001b[0mdenom\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmax_exp_avg_sqs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_correction2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 94\u001b[0;31m             \u001b[0mdenom\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mexp_avg_sq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_correction2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     95\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     96\u001b[0m         \u001b[0mstep_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlr\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mbias_correction1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x4C64xZ1UrQA",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_ddpg.save('/content/trained_models/trained_ddpg.zip')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPEXBcm-uBJo"
      },
      "source": [
        "### Model 4: **SAC**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HaWf2QeiLqyO",
        "outputId": "054fdbb0-dee8-4595-f7e4-767eb16d9002",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "SAC_PARAMS = {\n",
        "    \"batch_size\": 128,\n",
        "    \"buffer_size\": 100000,\n",
        "    \"learning_rate\": 0.0003,\n",
        "    \"learning_starts\": 100,\n",
        "    \"ent_coef\": \"auto_0.1\",\n",
        "}\n",
        "\n",
        "model_sac = agent.get_model(\"sac\",model_kwargs = SAC_PARAMS)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'batch_size': 128, 'buffer_size': 100000, 'learning_rate': 0.0003, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cpu device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ZgYVPqtKLvi3",
        "outputId": "08d13796-1f44-43e4-a2db-f6dc5119e9cb",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_sac = agent.train_model(model=model_sac, \n",
        "                             tb_log_name='sac',\n",
        "                             total_timesteps=50000)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4492284.431533596\n",
            "Sharpe:  0.791171424679218\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4610952.098005918\n",
            "Sharpe:  0.795419623984216\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4611061.021678517\n",
            "Sharpe:  0.7954298227533135\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4611200.059848312\n",
            "Sharpe:  0.7954511016078325\n",
            "=================================\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    episodes        | 4         |\n",
            "|    fps             | 17        |\n",
            "|    time_elapsed    | 653       |\n",
            "|    total_timesteps | 11572     |\n",
            "| train/             |           |\n",
            "|    actor_loss      | -1.06e+08 |\n",
            "|    critic_loss     | 1.04e+12  |\n",
            "|    ent_coef        | 3.26      |\n",
            "|    ent_coef_loss   | -191      |\n",
            "|    learning_rate   | 0.0003    |\n",
            "|    n_updates       | 11471     |\n",
            "|    reward          | 4611200.0 |\n",
            "----------------------------------\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4611486.965175941\n",
            "Sharpe:  0.7954869245190032\n",
            "=================================\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-30-752246e40d5e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m trained_sac = agent.train_model(model=model_sac, \n\u001b[1;32m      2\u001b[0m                              \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'sac'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m                              total_timesteps=50000)\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/finrl/drl_agents/stablebaselines3/models.py\u001b[0m in \u001b[0;36mtrain_model\u001b[0;34m(self, model, tb_log_name, total_timesteps)\u001b[0m\n\u001b[1;32m    103\u001b[0m             \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 105\u001b[0;31m             \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTensorboardCallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    106\u001b[0m         )\n\u001b[1;32m    107\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/sac/sac.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    298\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    299\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 300\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    301\u001b[0m         )\n\u001b[1;32m    302\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/off_policy_algorithm.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    369\u001b[0m                 \u001b[0;31m# Special case when the user passes `gradient_steps=0`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    370\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mgradient_steps\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 371\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgradient_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    372\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    373\u001b[0m         \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_training_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/sac/sac.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, gradient_steps, batch_size)\u001b[0m\n\u001b[1;32m    261\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m             \u001b[0mactor_loss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 263\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m             \u001b[0;31m# Update target networks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     86\u001b[0m                 \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                     \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py\u001b[0m in \u001b[0;36mdecorate_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     26\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    142\u001b[0m                    \u001b[0mlr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'lr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m                    \u001b[0mweight_decay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'weight_decay'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m                    eps=group['eps'])\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py\u001b[0m in \u001b[0;36madam\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps)\u001b[0m\n\u001b[1;32m     96\u001b[0m         \u001b[0mstep_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlr\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mbias_correction1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     97\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 98\u001b[0;31m         \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddcdiv_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexp_avg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdenom\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mstep_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     99\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-nUIx62dUvfF",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_sac.save('/content/trained_models/trained_sac.zip')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iidB5E27dfzh"
      },
      "source": [
        "### Model 5: **TD3**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XtRp1mWydkvs",
        "outputId": "90c04c04-2793-4bfc-f58c-b0df1431e061",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "TD3_PARAMS = {\"batch_size\": 100, \n",
        "              \"buffer_size\": 1000000, \n",
        "              \"learning_rate\": 0.001}\n",
        "\n",
        "model_td3 = agent.get_model(\"td3\",model_kwargs = TD3_PARAMS)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.001}\n",
            "Using cpu device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 553
        },
        "id": "argM0DtodmNL",
        "outputId": "3028f13d-2e3f-429b-87d4-ca998fb44057",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_td3 = agent.train_model(model=model_td3, \n",
        "                             tb_log_name='td3',\n",
        "                             total_timesteps=30000)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4473275.058447446\n",
            "Sharpe:  0.802631981256349\n",
            "=================================\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4420123.524870704\n",
            "Sharpe:  0.7803586399583237\n",
            "=================================\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-32-eb80aaa714a9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m trained_td3 = agent.train_model(model=model_td3, \n\u001b[1;32m      2\u001b[0m                              \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'td3'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m                              total_timesteps=30000)\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/finrl/drl_agents/stablebaselines3/models.py\u001b[0m in \u001b[0;36mtrain_model\u001b[0;34m(self, model, tb_log_name, total_timesteps)\u001b[0m\n\u001b[1;32m    103\u001b[0m             \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 105\u001b[0;31m             \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTensorboardCallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    106\u001b[0m         )\n\u001b[1;32m    107\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/td3/td3.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    211\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         )\n\u001b[1;32m    215\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/off_policy_algorithm.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    369\u001b[0m                 \u001b[0;31m# Special case when the user passes `gradient_steps=0`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    370\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mgradient_steps\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 371\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgradient_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    372\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    373\u001b[0m         \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_training_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/td3/td3.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, gradient_steps, batch_size)\u001b[0m\n\u001b[1;32m    168\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcritic\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    169\u001b[0m             \u001b[0mcritic_loss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 170\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcritic\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    171\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    172\u001b[0m             \u001b[0;31m# Delayed policy updates\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     86\u001b[0m                 \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                     \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py\u001b[0m in \u001b[0;36mdecorate_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     26\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    142\u001b[0m                    \u001b[0mlr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'lr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    143\u001b[0m                    \u001b[0mweight_decay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'weight_decay'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 144\u001b[0;31m                    eps=group['eps'])\n\u001b[0m\u001b[1;32m    145\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py\u001b[0m in \u001b[0;36madam\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps)\u001b[0m\n\u001b[1;32m     84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m         \u001b[0;31m# Decay the first and second moment running average coefficient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 86\u001b[0;31m         \u001b[0mexp_avg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbeta1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     87\u001b[0m         \u001b[0mexp_avg_sq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbeta2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddcmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconj\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mamsgrad\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nGXdd4uEUzCk",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trained_td3.save('/content/trained_models/trained_td3.zip')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2Ma6YpTlnuZ"
      },
      "source": [
        "## Trading\n",
        "Assume that we have $1,000,000 initial capital at 2019-01-01. We use the DDPG model to trade Dow jones 30 stocks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uas8U6k455sI",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trade = data_split(df,'2020-07-01', '2021-10-31')\n",
        "e_trade_gym = StockPortfolioEnv(df = trade, **env_kwargs)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LcGYlhyal205",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trade.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qq4W9FbSstT7",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df_daily_return, df_actions = DRLAgent.DRL_prediction(model=trained_a2c,\n",
        "                        environment = e_trade_gym)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uJvj3pXt_Ukg",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df_daily_return.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vXMMG_9SdKTu",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df_daily_return.to_csv('df_daily_return.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tByVcZ2L9TAJ",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df_actions.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xBX3Y68o1vRG",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df_actions.to_csv('df_actions.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFO42LcomPUT"
      },
      "source": [
        "<a id='6'></a>\n",
        "# Part 7: Backtest Our Strategy\n",
        "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fAvxipWFmUe8"
      },
      "source": [
        "<a id='6.1'></a>\n",
        "## 7.1 BackTestStats\n",
        "pass in df_account_value, this information is stored in env class\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1oGu3PCa8l6L",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "from pyfolio import timeseries\n",
        "DRL_strat = convert_daily_return_to_pyfolio_ts(df_daily_return)\n",
        "perf_func = timeseries.perf_stats \n",
        "perf_stats_all = perf_func( returns=DRL_strat, \n",
        "                              factor_returns=DRL_strat, \n",
        "                                positions=None, transactions=None, turnover_denom=\"AGB\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hqvwr6SY8l9A",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "print(\"==============DRL Strategy Stats===========\")\n",
        "perf_stats_all"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XcWzfa6UloDM",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "#baseline stats\n",
        "print(\"==============Get Baseline Stats===========\")\n",
        "baseline_df = get_baseline(\n",
        "        ticker=\"^DJI\", \n",
        "        start = df_daily_return.loc[0,'date'],\n",
        "        end = df_daily_return.loc[len(df_daily_return)-1,'date'])\n",
        "\n",
        "stats = backtest_stats(baseline_df, value_col_name = 'close')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lVVCMVSAmcrI"
      },
      "source": [
        "<a id='6.2'></a>\n",
        "## 7.2 BackTestPlot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2fqSEF5PfjjT",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "import pyfolio\n",
        "%matplotlib inline\n",
        "\n",
        "baseline_df = get_baseline(\n",
        "        ticker='^DJI', start=df_daily_return.loc[0,'date'], end='2021-07-01'\n",
        "    )\n",
        "\n",
        "baseline_returns = get_daily_return(baseline_df, value_col_name=\"close\")\n",
        "\n",
        "with pyfolio.plotting.plotting_context(font_scale=1.1):\n",
        "        pyfolio.create_full_tear_sheet(returns = DRL_strat,\n",
        "                                       benchmark_rets=baseline_returns, set_context=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-2DgsIW-fh0s",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "te3Ibcj5hUbz"
      },
      "source": [
        "## Min-Variance Portfolio Allocation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1VE2eUEuhMKs",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "!pip install PyPortfolioOpt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i0NiefM7hHn0",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "from pypfopt.efficient_frontier import EfficientFrontier\n",
        "from pypfopt import risk_models"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lDYDIBH9hcUP",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "unique_tic = trade.tic.unique()\n",
        "unique_trade_date = trade.date.unique()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8qvk_pJ4iV32",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EICNukJZgnWl",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "#calculate_portfolio_minimum_variance\n",
        "portfolio = pd.DataFrame(index = range(1), columns = unique_trade_date)\n",
        "initial_capital = 1000000\n",
        "portfolio.loc[0,unique_trade_date[0]] = initial_capital\n",
        "\n",
        "for i in range(len( unique_trade_date)-1):\n",
        "    df_temp = df[df.date==unique_trade_date[i]].reset_index(drop=True)\n",
        "    df_temp_next = df[df.date==unique_trade_date[i+1]].reset_index(drop=True)\n",
        "    #Sigma = risk_models.sample_cov(df_temp.return_list[0])\n",
        "    #calculate covariance matrix\n",
        "    Sigma = df_temp.return_list[0].cov()\n",
        "    #portfolio allocation\n",
        "    ef_min_var = EfficientFrontier(None, Sigma,weight_bounds=(0, 0.1))\n",
        "    #minimum variance\n",
        "    raw_weights_min_var = ef_min_var.min_volatility()\n",
        "    #get weights\n",
        "    cleaned_weights_min_var = ef_min_var.clean_weights()\n",
        "    \n",
        "    #current capital\n",
        "    cap = portfolio.iloc[0, i]\n",
        "    #current cash invested for each stock\n",
        "    current_cash = [element * cap for element in list(cleaned_weights_min_var.values())]\n",
        "    # current held shares\n",
        "    current_shares = list(np.array(current_cash)\n",
        "                                      / np.array(df_temp.close))\n",
        "    # next time period price\n",
        "    next_price = np.array(df_temp_next.close)\n",
        "    ##next_price * current share to calculate next total account value \n",
        "    portfolio.iloc[0, i+1] = np.dot(current_shares, next_price)\n",
        "    \n",
        "portfolio=portfolio.T\n",
        "portfolio.columns = ['account_value']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S2zmv1ISkpXu",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "portfolio.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5RmNa7m_f590",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "a2c_cumpod =(df_daily_return.daily_return+1).cumprod()-1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y821cLKkhCn6",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "min_var_cumpod =(portfolio.account_value.pct_change()+1).cumprod()-1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5E1X9FFGgqeZ",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "dji_cumpod =(baseline_returns+1).cumprod()-1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9g4VpoqlMqR",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ih6Jim-blNKY"
      },
      "source": [
        "## Plotly: DRL, Min-Variance, DJIA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CJRH-FX3hTRZ",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "from datetime import datetime as dt\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import plotly\n",
        "import plotly.graph_objs as go"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mpfgJcG5PInj",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "time_ind = pd.Series(df_daily_return.date)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CM-NJKa8g7Jp",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "trace0_portfolio = go.Scatter(x = time_ind, y = a2c_cumpod, mode = 'lines', name = 'A2C (Portfolio Allocation)')\n",
        "\n",
        "trace1_portfolio = go.Scatter(x = time_ind, y = dji_cumpod, mode = 'lines', name = 'DJIA')\n",
        "trace2_portfolio = go.Scatter(x = time_ind, y = min_var_cumpod, mode = 'lines', name = 'Min-Variance')\n",
        "#trace3_portfolio = go.Scatter(x = time_ind, y = ddpg_cumpod, mode = 'lines', name = 'DDPG')\n",
        "#trace4_portfolio = go.Scatter(x = time_ind, y = addpg_cumpod, mode = 'lines', name = 'Adaptive-DDPG')\n",
        "#trace5_portfolio = go.Scatter(x = time_ind, y = min_cumpod, mode = 'lines', name = 'Min-Variance')\n",
        "\n",
        "#trace4 = go.Scatter(x = time_ind, y = addpg_cumpod, mode = 'lines', name = 'Adaptive-DDPG')\n",
        "\n",
        "#trace2 = go.Scatter(x = time_ind, y = portfolio_cost_minv, mode = 'lines', name = 'Min-Variance')\n",
        "#trace3 = go.Scatter(x = time_ind, y = spx_value, mode = 'lines', name = 'SPX')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "35nVVmEuhGa1",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        "fig = go.Figure()\n",
        "fig.add_trace(trace0_portfolio)\n",
        "\n",
        "fig.add_trace(trace1_portfolio)\n",
        "\n",
        "fig.add_trace(trace2_portfolio)\n",
        "\n",
        "\n",
        "\n",
        "fig.update_layout(\n",
        "    legend=dict(\n",
        "        x=0,\n",
        "        y=1,\n",
        "        traceorder=\"normal\",\n",
        "        font=dict(\n",
        "            family=\"sans-serif\",\n",
        "            size=15,\n",
        "            color=\"black\"\n",
        "        ),\n",
        "        bgcolor=\"White\",\n",
        "        bordercolor=\"white\",\n",
        "        borderwidth=2\n",
        "        \n",
        "    ),\n",
        ")\n",
        "#fig.update_layout(legend_orientation=\"h\")\n",
        "fig.update_layout(title={\n",
        "        #'text': \"Cumulative Return using FinRL\",\n",
        "        'y':0.85,\n",
        "        'x':0.5,\n",
        "        'xanchor': 'center',\n",
        "        'yanchor': 'top'})\n",
        "#with Transaction cost\n",
        "#fig.update_layout(title =  'Quarterly Trade Date')\n",
        "fig.update_layout(\n",
        "#    margin=dict(l=20, r=20, t=20, b=20),\n",
        "\n",
        "    paper_bgcolor='rgba(1,1,0,0)',\n",
        "    plot_bgcolor='rgba(1, 1, 0, 0)',\n",
        "    #xaxis_title=\"Date\",\n",
        "    yaxis_title=\"Cumulative Return\",\n",
        "xaxis={'type': 'date', \n",
        "       'tick0': time_ind[0], \n",
        "        'tickmode': 'linear', \n",
        "       'dtick': 86400000.0 *80}\n",
        "\n",
        ")\n",
        "fig.update_xaxes(showline=True,linecolor='black',showgrid=True, gridwidth=1, gridcolor='LightSteelBlue',mirror=True)\n",
        "fig.update_yaxes(showline=True,linecolor='black',showgrid=True, gridwidth=1, gridcolor='LightSteelBlue',mirror=True)\n",
        "fig.update_yaxes(zeroline=True, zerolinewidth=1, zerolinecolor='LightSteelBlue')\n",
        "\n",
        "fig.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nXd-N5Rmh6H7",
        "pycharm": {
          "is_executing": true
        }
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}